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Virtually a Sisterhood: 
Social Connectedness and Online Collaboration 


Dr. George Banks-Weston 
Dr. Tammi Kolski 
Curriculum and Instruction 
College of Education 
University of South Carolina 
Columbia, South Carolina 


Abstract 

The purpose of this ethnographic research was to uncover the experiences of culturally diverse 
women participating in an online business strategy course and to evaluate the impact of virtual 
collaboration on social connectedness among this group. This study was situated within a 
national organization for women of color who are entrepreneurs. Data was collected from three 
members during an eight-week online course that also functioned as a virtual community of 
practice. Qualitative data was collected in three phases of semi-structured interviews with each 
participant being interviewed at the beginning, midpoint, and end of the course. Inductive 
analysis revealed that virtual collaboration positively impacted participants! feelings of social 
connectedness when activities were supported by use of virtual collaboration tools. The virtual 
collaboration tools allowed participants to work towards common goals, build a support network, 
and participate in skill development that contributed towards their career advancement. 

Keywords: social connectedness, virtual community of practice, virtual collaboration 
tools, social capital, and women of color 


Virtually a Sisterhood: Social Connectedness and Online Collaboration 

Professional women of color often encounter discrimination and marginalization that 
negatively impacts their careers (Neville et al., 2018; Scott & Hussein, 2019). Additionally, 
based on their status as double minorities, these women also experience threats of being 
stereotyped which can leave them feeling isolated (Alfred et al., 2019; Johnson et al., 2017). 
Among the challenges faced by these individuals is the lack of access to support systems that 
provide them opportunities for mentorship, networking, and professional development 
opportunities (Ong et al., 2018; Rice, 2017). Research has shown that when these women have 
access to resources, networks, and spaces where they feel supported, understood and connected, 
it has a positive impact on their personal and career success (Kumi-Yeboah et al., 2017; 
McLoughlin et al., 2018). 

The purpose of this study was to evaluate the impact of virtual collaboration on social 
connectedness among a group of culturally diverse women participating in an online professional 
development course for entrepreneurs. Social connectedness is identified as a factor in the 
success of underrepresented populations and past research has expressed a need for more studies 
centered on the professional experiences of these individuals (Ong et al., 2018). Specifically, the 
research question that guided this study was how does virtual collaboration impact social 
connectedness among a group of culturally diverse women participating in an online business 
strategy course for entrepreneurs. 

Conceptual Framework 

Social capital theory holds that social constructs impact how one views their role in a 
group (Bourdieu, 2011). Current research on social capital evolves this theory, identifying social 
capital as a construct that can be directly aligned to the social and economic well-being of 
individuals who belong to groups, networks, or communities (Waller et al, 2011; Zhang et al., 
2011). Waller et al. (2011) highlighted that the degree of social capital’s influence can be 
connected to an individual’s perception of belonging within these groups, and how this 
perception impacts their connection to other individuals in this group. There is value connected 
to an individual’s sense of belonging and subsequently their social connectedness to a group or 
network, with these two components often identified as the two major tenets of social capital 
theory (Waller et al., 2011). 

When an individual has a strong sense of belonging it means they feel fully integrated 
into their environment and as such, can maximize opportunities to benefit from the resources of 
that environment. An internal form of sense of belonging is social connectedness (Costen et al., 
2013). Social connectedness is centered on the opinion of one’s self in relation to other people 
within a group or network and represents the emotional distance between one’s self and other 
people within their network (Paolucci et al., 2021). Social connectedness is also viewed as the 
strength of the relationships that an individual has with others, and how these relationships 
influence their interactions with others in a particular network. 

Social Connectedness and Underrepresented Populations 

As it relates to underrepresented populations, Costen et al. (2013) revealed that social 
connectedness can be an influencer in an individual’s ability to acclimate to an environment. 
Aligning with this study, Museus and Saelua (2017) highlighted that more culturally engaged 
climates can positively impact feelings of connectedness and belonging among 
underrepresented individuals in learning environments. In these environments, individuals 
from underrepresented populations seek out or build their own support systems (Ong et al., 
2018; West, 2017, 2019). Participation in these types of support networks, also known as 


counterspaces, has been shown to lessen feelings of isolation by helping women form 
connections with others who may also have had similar challenges with discrimination and 
marginalization (Johnson et al., 2017; Ong et al., 2018; West, 2017, 2019). 

A high degree of social connectedness in learning environments can also have 
implications for an individual’s success (Mishra, 2020; Museus & Saelua, 2017). Factors that 
positively influence social connectedness in learning environments are those that intentionally 
embrace diversity and inclusion as well as depth and quality of relationships with peers who 
share similar ethnic backgrounds (Costen et al., 2013), and support services such as 
counseling, coaching, or mentorship (Mishra, 2020). 

With the advancement of technology, recent research has explored the role technology 
plays in building social capital in online environments. Much of the research showcases the ways 
in which individuals engage or leverage technological tools within these environments to engage 
with others, to find and create communities, and to learn in ways that impact or help individual’s 
form their social identities and acquire social capital (Grottke et al., 2018; Roldan et al., 2017). 
As such, online interactions have implications on an individual’s feelings of connectedness to 
other individuals in those networks. Additionally, past research on building social capital in an 
online forum uncovered how the use of technology, namely computer mediated communication 
tools, both supports and interferes with an individual’s feelings of connectedness (Roldan et al., 
2017). 

In in-person environments an individual’s feelings of social connectedness can be 
assessed through their perceptions of their position or belonging in that group, how they feel they 
demonstrate the predictable behaviors of that group, and the physical traits of the environment 
(Callahan et al., 2015; Irgens, 2019). However, in online environments, each of these aspects are 
more challenging to observe (Slagter van Tryon & Bishop, 2009). Further suggesting that 
enhanced feelings of social connectedness in online environments are closely tied to activities 
and behaviors that support collaboration, open communication, and provide opportunities to 
identify common goals or to share resources (Grottke et al., 2018; Mays, 2016). When these 
strategies and activities are leveraged in online environments and strong feelings of social 
connectedness are able to develop, it can be an influencer in career development, persistence, 
and achievement (Donelan, 2016; Heidari et al., 2020). These opportunities are often presented 
through the use of online social networking tools, where individuals can build formal or informal 
networks (Donelan, 2016; Heidari et al., 2020; Roldan et al., 2017). Through these networks, 
individuals can contribute their own knowledge and at the same time, learn from others, and gain 
access to career opportunities. 

Virtual Communities of Practice & Virtual Collaboration 

Lave and Wegner (1991) defines communities of practice as systems or networks where 
members develop a shared understanding about who they are, what they are doing, and how each 
of these components apply to them individually as well as within a collective community. Virtual 
communities of practice (VCoPs) are defined as communities where members build, share, and 
create knowledge in an online environment and have been heralded as vital to collective learning 
in a society that is more reliant on technology (Ardichvili et al., 2003; Ardichvili, 2008). 
Through VCoPs, those individuals who are novices can build knowledge and acquire resources 
from more experienced individuals (known as experts) participating in that community (Hafeez 
et al., 2019; Hernandez-Sotoa et al., 2021). 

Research of Callahan et al. (2015), Liu et al. (2017), and West (2019) suggested to 
combat underrepresentation and improve the retention of diverse talent in professional industries, 


activities that enhance social connectedness must be integrated into virtual collaboration 
strategies which are essential to the success of a VCoP. Although definitions of virtual 
collaboration are varied, it is most commonly defined as activities or acts in a virtual 
environment centered around a common goal, purpose, or task (Taras et al., 2013). For purposes 
of this study, virtual collaboration was defined as a group of individuals working toward 
common goals in an online professional development environment. These common goals were 
supported through the use of audio-conferencing, videoconferencing, or computer-mediated 
technologies also known as virtual collaboration tools (Poppe et al., 2017). Several virtual 
collaboration strategies have been identified as relevant to knowledge sharing and learning in 
VCoPs. These strategies stress the importance of creating inclusive environments where all 
members feel they can actively participate, even though their participation is often framed as 
voluntary (Ardichvili, 2008; McLoughlin et al., 2018). Porter et al. (2011) identified that VCoPs 
are more successful in driving participation from members when those who belong to that 
community have role clarity and defined responsibilities. This can also include opportunities for 
members who are experts to provide mentorship to others (Hernandez-Sotoa et al., 2021) as well 
as find ways to disperse knowledge creation equally across members (Barnett et al., 2016). In 
addition, Hernandez-Sotoa et al. (2021) holds that members of the community must not only 
understand what their role is, but also how it aligns with the greater mission, values, and goals of 
that VCoP. 
Methods 

Approaching the research through an ethnographic lens, this study was conducted 
through The Prominence Association for Women (a pseudonym), a national membership 
organization for professional women. Founded in 2012, The Prominence Association for 
Women was founded to provide professional coaching, mentorship, and educational resources 
for women of color who were seeking to grow their career or their business. As a part of their 
membership in the organization, women benefited from access to a suite of online self-paced 
courses, one-on-one coaching opportunities with the organization’s founders, and invitations to 
participate in specialized career and professional development programs known as Mastermind 
Sessions. Mastermind Sessions take place over an eight-week period at least four times per 
year. Each course is guided by a different theme and are designed to provide members of the 
organization opportunities to quickly develop and execute specific strategies for their business 
or to hone a specific set of skills related to growing as professionals. The topic of the 
Mastermind Session that underpinned this research focused on developing a growth and 
customer engagement strategy for an online business. The course was conducted entirely online 
through a mix of weekly one-to-two-hour workshops led by leaders of The Prominence 
Association for Women or industry experts, and involved developing and presenting a final 
project or presentation to the leadership team. The five free virtual collaboration tools available 
for the women to use in completing their final project were Zoom, Facebook Groups, Facebook 
Messenger, E-mail, and Google Drive. In addition to the weekly workshops, each week 
participants were assigned an accountability partner, who was one of their peers in the course. 
The Mastermind Session referenced in this study was viewed as a VCoP because all interactions 
occurred online and virtual collaboration tools were used to stimulate feelings of social 
connectedness. Being an entrepreneur and marketing professional of color, the lead researcher 
of this study served as a board member of The Prominence Association for Women to advance 
equality for the underrepresented female membership of this organization. The lead researcher 
also served as a facilitator of other Mastermind Sessions; however, was not the facilitator of the 


professional development session for this study. While the lead researcher shared similar 
cultural standpoints, professional predispositions, and personal experiences of this study’s 
participants, their internal positions did not influence the outcomes. The second author, who did 
not share characteristics of the study’s participants, assured that bias or influenced inquiries 
were removed. 

A purposive sampling (Etikan & Bala, 2017) of Mastermind course participants 
allowed for factors of race/ethnicity, age, educational background, profession, and tenure in the 
organization to be considered. Members participating in the online business strategy course who 
were with the Prominence Association for Women for less than a year were excluded from this 
study. At the beginning of the online business strategy course, an e-mail was sent to the nine 
potential participants who had enrolled in the professional development course asking for their 
consent to participate in the study. The three women who responded and provided consent to 
participate in this study offered culturally diverse backgrounds and had varied levels of 
experience with technology. Table 1 provides a description of each participant, aligned to their 
pseudonym, and includes their age, race, tenure of membership, education, geographic location 
and entrepreneurship business type. 


Table 1 
Participant DescrpHons = ——— Se  _ 
Pseudonym Tenure of 
(She/Her/Hers) Age Race membership Education Location Business 
Real Estate / 
Jordan 30's Black 3 Years Master Degree New Jersey Travel 
Floral Design / 
Louisa 40's Hispanic 1.5 Years Bachelor Degree New Jersey Event Planning 
Non-Profit 
Maisie 50's Hispanic 1 Year Bachelor Degree Pennsylvania Scholarship Fund 


Data Collection and Data Analysis 

Ethnography, a qualitative research method, was used to deeply understand the social 
and cultural life of the women in this study (Glesne, 2016). The qualitative data was collected 
from a three phase semi-structured interview protocol that produced thick written cultural 
descriptions of the women's experiences throughout the course. Interviews took place at the 
initiation, midpoint, and at the conclusion of the course, each lasting 30 - 45 minutes in 
duration. Because participants were geographically dispersed, interviews were conducted via 
Zoom Meeting with the lead researcher. Methods for rigor and trustworthiness included 
triangulation, member checking, and weekly peer debriefing with the co-researcher (Mertler, 
2017). Member checking was a multistep process that began with gaining participants 
feedback by providing them a presentation consisting of data collected from interviews, 
analytical memos, and researcher notes. Next, an individualized report of their specific 
responses was sent via e-mail to each woman who participated in the study. Finally, a 30- 
minute member check discussion was scheduled via Zoom Meeting with each participant to 
review their feedback and ensure their experiences and words were captured correctly. Each of 
the three participants validated their semi-structured interview transcripts to accurately capture 
their interaction and offered no feedback that changed the findings of the qualitative analysis. 


Utilization of Delve coding software assisted in an inductive analysis of the nine 
transcribed interviews. Saldafia’s (2016) first and second cycle coding techniques (Structural, 
In Vivo, Process, Value, Pattern, and Focus Coding) resulted in 224 codes, eight categories, 
three themes and one assertion: Participants perceived technology can support the development 
of strong intimate relationships when entrepreneurs who are women share similar 
backgrounds, common goals, and past experiences. 

Findings 

The findings of this study illustrate that the participants' use of virtual collaboration tools 
had an impact on their personal growth and skill development throughout the course, the ways in 
which they interacted with others in the course, as well as the authentic connections and bonds 
they were able to build as a result of their participation. Participants indicated that skill 
development and relationship building were enhanced by synchronous one-on-one and group 
interaction that allowed them to engage in discussion, provide feedback to each other, and offer 
guidance in real time. Participants expressed those activities that supported these types of 
interactions also lessened the feelings of isolation they feel as women of color who are pursuing 
entrepreneurship. Additionally, the activities created a learning environment where they felt 
supported and resulted in them identifying a positive impact on their personal and career success. 
These findings led to the assertion that participants perceived the use of technology can support 
the development of strong intimate relationships between women entrepreneurs who share 
similar backgrounds, common goals, and past experiences. These findings align with the existing 
body of research on VCoPs that has revealed technology supported activities are essential to an 
individual’s social connectedness and subsequently their engagement with other members of that 
community (Ardichvili, 2008; Hafeez et al., 2019; McLoughlin et al., 2018). As well, the 
findings also align with research on counterspaces and women of color, where social 
connectedness is identified as an indicator in one’s ability to develop relationships, and as such, 
garner benefits from those relationships that are built with others in that counterspace (Johnson et 
al., 2017; Ong et al., 2018; West, 2019). Support for the assertion and the existing body of 
research was illustrated through the three themes that emerged out of the qualitative data 
collected: (a) Entrepreneurial Progression, (b) Richness of Synchronous Interaction and (c) 
Interdependence Fosters Authentic Connections. 
Theme 1: Entrepreneurial Progression 

Entrepreneurial progression centered on how an individual’s personal history, 
background, and experiences as women of color who are professionals affected their 
participation in the online business strategy course. This included aspects of the course that 
impacted professional skill development and learning outcomes. Research suggested that 
entrepreneurs who are women often experience discrimination and marginalization in ways that 
can leave them feeling isolated or unsupported in their careers (Block et al., 2019; Callahan et 
al., 2015; Wilkins-Yel et al., 2019). Research has also shown that being provided opportunities 
to connect with others who share similar background and professional experiences can lessen 
feelings of isolation (Ong et al., 2018; Vaccaro et al., 2019), and can help them to develop and 
hone skills necessary to succeed in their careers (Johnson et al., 2017; Ong et al., 2018; West, 
2017, 2019). In this study, entrepreneurial progression was categorized by how an individual’s 
identity as a woman of color shaped their views as an entrepreneur, the activities these women 
felt contributed to their skill development, and the specific outcomes and impacts to their career 
development they described as a result of their participation. 


The three participants in the study - Louisa, Maise, and Jordan - highlighted that they 
were attracted to becoming members of The Prominence Association for Women because its 
focus was about women of color who are entrepreneurs. Each of these women shared that they 
experienced feelings of loneliness, lacked a connection with like-minded female business owners 
who shared similar backgrounds, and had a desire for the support and encouragement that The 
Prominence Association for Women offered them. In fact, the desire to connect and to 
collaborate with other women of color was one of the leading reasons each participant was 
attracted to this organization. All participants described that as women of color, they had not 
encountered organizations like The Prominence Association for Women that allowed them to 
connect with like-minded business owners who share similar backgrounds. For example, Jordan 
in her initial interview stated, “I hadn't heard of a membership organization that was specifically 
for entrepreneurs who are women of color...that was founded by a Black woman... and that was 
really fostering this environment of collaboration and learning and growing together.” This 
statement was similar to one by Louisa, who highlighted in her final interview that she felt less 
isolated and alone as a result of taking the course. Louisa said, “Being an entrepreneur is a very 
lonely journey...but then I was able to find another chick out there that has the same struggle. It 
doesn't have to be the same business, but she's living her own, same, lonely race.” 

While the desire to connect with other women of color who were entrepreneurs and 
building a support network was one of the main attractors to the organization, participants 
expressed that they enrolled in the online business strategy course to cultivate skills that they felt 
would aid them in their future success as entrepreneurs. At the onset of the course, Louisa and 
Maisie, expressed a lack of self confidence in the skills they felt were essential in their roles as 
entrepreneurs. These skills included better understanding how to market their businesses, how to 
communicate with customers, how to gain funding and sponsorships, and how to expand their 
products or services. However, as time moved forward in the course, each of the three 
participants expressed feelings of appreciation, confidence, and competence as it related to their 
skill development. Jordan stated in her final interview, “I had tried using these [content] creation 
tools on my own. [After the course] I was like, wow, I can’t believe I did that compared to what I 
was creating before.” By the conclusion of the study, all three participants indicated that they felt 
they had cultivated skills in social media, marketing and branding, and collaboration to a level 
where they were able to complete a variety of tasks on their own without assistance. Louisa 
expressed in her final interview, “Before I was intimidated...now it’s like I can walk on my own, 
and I am a lot more tech savvy.” For communications skills, specifically, all participants 
commented that over the duration of the online business strategy course they were able to 
increase their capability to communicate with their peers and pitch their business to potential 
investors or donors. For example, Louisa stated in her midpoint interview, “I actually feel like 
I'm developing better communication skills because before I just didn't have the words [to 
explain] or I didn't have everything in order in my brain.” 

In the final interviews, each of the participants shared that through working alongside 
and receiving encouragement from the other women enrolled in the course, that they were able 
to learn how to use social media tools, video conferencing, and web design technologies as well 
as gain access to professional and financial resources that would benefit their business. 
Additionally, they discussed creating a stronger network of professionals through the 
relationships they were able to build with one another. Maisie noted in her final interview that 
through the help and coaching she received from other members in the course, she was able to 
learn about the different types of customer relationships she needed for her business to grow. 


Building the relationships that I did with everybody and how we helped each other, there 
was times that I would feel stuck and didn't know what to do and they really reached out 
to me and helped me out a lot. Um, that meant a lot. It still means a great deal to me and 
you know, learning about the different relationships, the customer relationships. 
These sentiments supported these findings that when women of color have access to 
counterspaces, they are able to gain support and mentorship that they may not have in 
mainstream forums. 
Theme 2: Richness of Synchronous Interaction 

Throughout the study the participants' experiences were explored based on how 
interactions occurred within a VCoP (Ardichvili, 2008; Hafeez et al., 2019). Research has shown 
that in VCoPs, interactions can happen using a multitude of virtual collaboration tools (Mather & 
Cummings, 2014; McLoughlin et al., 2018). The nature and use of these various technologies 
can have an impact on the value a member may glean from their participation in the VCoP 
(Altebarmarkian & Alterman, 2019; Barnett et al., 2012). In this study these tools supported 
various types of interactions including communication between members, sharing of 
information, and task completion. Additionally, participants were able to leverage these virtual 
collaboration tools for both synchronous and asynchronous interactions throughout the study. 

Participants engaged in synchronous interaction through the use of technologies such as 
Zoom, Facebook Messenger, and through text messaging and telephone calls. However, each of 
these tools served different purposes for participants based on their individual needs at the time. 
For example, when discussing Zoom, participants highlighted a number of key benefits including 
the opportunity for face-to-face communication (which participants felt were more personable) 
as well as capabilities such as screen and desktop sharing which allowed for other participants to 
provide feedback on their work. Participants also discussed the value of Zoom breakout rooms 
for engagement in personalized small group discussions and activities. This supports research 
that using these synchronous tools with a collaborative purpose can often create more value in 
VCoP (McLoughlin et al., 2018; Wang & Huang, 2018). This value was expressed by Louisa 
who identified how the breakout rooms offered an opportunity for others to ask her questions or 
for her to express opinions that she may have felt uncomfortable asking in the larger group. 
Louisa stated in her midpoint interview, 

Let's say maybe a person who's more of an introvert may feel intimidated because I'm an 

extrovert and I got all these questions, the breakout room helped.... If I'm by myself with 

someone who I feel is at a plateau, then I will start asking questions...reiterating things 
we ve learned in the course. 

Outside of the weekly workshops facilitated through Zoom, participants used Facebook 
Messenger, text messaging, and phone calls to communicate with their assigned accountability 
partners and to get assistance from their peers regarding their businesses. For example, Maisie 
expressed a point where she was trying to find funding for her business while completing her 
final presentation and how she was able to leverage text messaging to accomplish this. She stated 
in her midpoint interview, “I texted my accountability partner to ask her about different funds 
that I wasn't aware of and she helped me with that...” 

In addition to synchronous interactions, virtual collaboration tools can also support 
asynchronous interactions and have been found helpful in document editing and feedback, 
formal or informal discussions, or sharing resources (Antoci et al., 2012; Porter et al., 2011). In 
this study, the asynchronous interaction occurred primarily through Facebook Groups. Aligning 
with research on computer mediated communication such as discussion boards and information 


sharing tools (Antoci et al., 2012; Kabilan, 2016), Facebook Groups functioned as a repository of 
questions and where resources were shared over the duration of the eight-week online business 
strategy course. Additionally, participants took advantage of Facebook Groups to engage their 
peers when they needed assistance, to ask questions that they felt would be beneficial to others, 
to provide encouragement, when they had news or information to share, or to keep up to date on 
content they may have missed in one of the weekly workshops. For example, Maisie noted in her 
final interview, “In the Facebook Group itself I would post questions or concerns if there was 
anything going on that I wasn't too sure of or I would help them out and encourage them.” 
Similarly, Jordan highlighted that Facebook Groups provided her an opportunity to share 
resources and information that could benefit her peers. In her final interview she provided an 
example, “TI literally went to a training and then straight from the training... wanted to share the 
information with [my classmates] ...so I posted it in the Facebook Group.” 

Although other technologies such as E-mail, Instagram, and Canva were mentioned, as it 
related to asynchronous interaction they were used sparingly. In instances where they were used, 
they served as a way to share information or to ask a question, however, as already identified 
Facebook Groups was the preferred method of communication and asynchronous interaction. 
Theme 3: Interdependence Fosters Authentic Connections 

When an individual has a strong sense of connection to others in a specific organization 
or group, it can positively impact their personal and professional success (Virick & Greer, 2012; 
Yoon et al., 2012). During the eight-week online business strategy course used for this research, 
participants cited many opportunities they were given to build strong and authentic bonds and 
relationships. Relationships that they believed contributed to their success in the course, but also 
aided them in furthering their businesses. The theme of Interdependence Fosters Authentic 
Connections focused on the behaviors and activities that fostered these connections and how 
those behaviors and activities created feelings of interdependence among the participants. There 
were multiple occasions where the participants likened the relationships, they were building with 
their peers to those they had with their family. This aligns with research on social connectedness 
which asserts that in order for someone to feel like they belong to a group they must feel some 
type of connectedness, affiliation, and companionship to others (Framke et al., 2019; Irgens, 
2019). Each of these feelings were expressed when participants discussed the bonds, they were 
building with others in their course. These bonds were initially described during the first weeks 
of the online business strategy course, and were revealed by all participants in both the initial and 
midpoint interviews. For example, Jordan, in her midpoint interview shared, ““We have a 
sisterhood, an environment, a safe place, for Black and Brown women to come together ... 
where our voices and frustrations can be heard. ... We can work together and give each other 
support through hard times.” 

Because women of color who are professionals have experienced discrimination over the 
course of their academic and professional lives, this also impacts how they acculturate into new 
VCoPs (Ardichvili, 2008; Barnett et al., 2012). These issues are often marked by feelings of 
doubt, apprehension and judgement which can prevent them from fully receiving the benefits 
that participation in such communities provide (Hernandez-Sotoa et al., 2021). Throughout this 
study, all three participants shared challenges they encountered or apprehensions they felt due to 
age, experience, and undeveloped business acumen and technology skills. For example, Maisie 
cited several examples where she felt uncomfortable, frustrated, or unprepared based on her lack 
of knowledge or capability to perform a task or participate in a discussion in comparison to her 
peers. However, by the end of the study each of the participants noted that being in the course 


helped them to become less intimidated when using technology, and that they were all able to 
utilize the other woman as resources. In her midpoint interview, Maisie shared, 

I got really frustrated...and I almost started to cry... because I [didn’t] know how to do a 

website ... and I felt alone. So, when [the facilitator] came to me [to describe my 

website] and I said to her, I can't do this...and I [left] because I was so upset...the next 
day, Jenny messages me this long message and she just made my day. She's like, I know 
she says, this is personal for you ... just don't give up. And you know that meant a lot. 

Similarly, when asked about her comfort with technology in her final interview Louisa 
stated, “I don’t feel intimidated. I don't know everything. I'm not an expert, but...with the ladies, 
we can figure it out. We can pull up the YouTube, show me how to do this... but before it would 
stop me.” 

Supporting each other and working towards a common goal was another aspect that 
participants believed had benefited them and their business. In her final interview, Jordan 
provided specific examples of how this occurred, 

I've learned a lot from the other women...I've learned how to be more of a financial 

steward and take control of your finances, how to deal with difficult people and how to 

work on dealing with that difficult customer .. and still providing the utmost customer 
service...I've learned different technologies. 
These outcomes were supported and described by Maisie and Louisa as well, and showcased 
how through collaborative work and mutual support, each of the participants were able to 
develop skills and garner resources from their participation in the online business strategy 
course. 
Implications 

With opportunities for support networks being scarce, women of color have often been 
forced to seek out or build their own support systems (Ong et al., 2018; West, 2017, 2019). 
Participation in these types of support networks, has been shown to lessen feelings of isolation 
helping these women form connections with others who may also have had similar challenges 
with discrimination and marginalization (Johnson et al., 2017; Ong et al., 2018; West, 2017, 
2019). The findings of this study suggest that virtual collaboration could offer these women more 
access to spaces that are free of the discrimination, sexism, marginalization, and oppression they 
face in their day-to-day lives. The findings also infer that when these virtual settings are guided 
by common goals and collaborative activities and are supported by the right choice of virtual 
collaboration tools, that women entrepreneurs are able to develop strong feelings of social 
connectedness. This social connectedness can aid in building authentic, intimate relationships 
with each other in ways that positively impact their confidence, help to hone and develop skills 
for their careers, and aid them in building a network of support for their professional success 
(Mishra, 2020; Museus & Saelua, 2017; Strdbel et al., 2017). 

In addition, the findings of this study support past research that integrating virtual 
collaboration tools into online learning environments can be essential to establish common goals, 
provide opportunities for synchronous interaction, and embed activities that allow learners to 
develop their technological aptitude (Mather & Cummings, 2014; Nistor et al., 2012). When 
learners come from underrepresented backgrounds the presence of these components can allow 
them to build confidence, and encourage relationship building with their peers (Mishra, 2020; 
Museus & Saelua, 2017). As with this study, the development of these relationships through 
virtual collaboration can impact social connectedness, and ultimately provide underrepresented 
learners who are professionals with resources, education, and the support they need to grow and 
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develop as entrepreneurs. When audio-conferencing, videoconferencing, or computer-mediated 
technologies are embedded in online professional development courses, social connectedness can 
be developed and can be an influencer in career development, persistence, and achievement 
(Donelan, 2016; Heidari et al., 2020). 
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Abstract 

Learning novel physical motor skills can be frustrating for some cadet students in the military 
movement class at the academy. A lack of clear feedback further exacerbates the problem for 
students’ skill development. This action research study examined the benefits of using video 
recording as augmented feedback to inform motor skill development in an applied gymnastics 
course at a service academy. Furthermore, it sought a more efficient method of providing female 
participants individualized feedback with the goal of decreasing the physical skill gap between 
the males and females at the academy. The primary aim was to identify the effects of video 
feedback on the instructor/student process for skill evaluation and skill improvement involving 
low female performers. The secondary aim was to examine the extent that giving video feedback 
promoted the ten female students’ motivation to learn, use of deliberate practice, autonomy, and 
competence. Data analysis revealed skill improvement on at least one skill event in the 
participants. Findings demonstrated that students perceived video feedback as an effective 
method for enhancing skill improvement in gymnastics class. These findings indicate video 
feedback can be used to improve motivation, deliberate practice, competence, and autonomy. 
The ability to visualize performance cues for the students may also result in faster motor skill 
acquisition. The study suggests video feedback is an effective method of augmented feedback for 
students struggling with novel physical motor skill progressions. 

Keywords: video feedback, augmented feedback, skill acquisition, female cadets, 
motivation, autonomy, competence 
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Benefits of Video Feedback on Low Performing Female Cadets in Physical Education: An 
Action Research Study 

Within the Physical Education Department at the academy, instructors are constantly 
seeking improvements and efficiencies within the curriculum and program of instruction. The 
core physical education classes are a graduation requirement. This means that every cadet at the 
academy must pass these classes to obtain a commission as an officer in the United States Army 
(Gist, 2016). The military movement class is one of the largest hurdles for the female cadets. 
However, this physically demanding course is a great indicator of future success throughout the 
physical program at the academy. As leaders, the academy is responsible for giving purpose, 
direction, and motivation to their students (Baghurst et al., 2015). One of the biggest challenges 
is motivating lowest performers to effectively use their practice time to improve their physical 
skills (Ellison & Woods, 2016). One of the greatest limiting factors to skill acquisition for 
students is the lack of quality feedback provided during instruction (Coelho, 2019; Turner & 
West, 2013). Performance based feedback is a key part to effective learning. The design of this 
study added to the existing sports and physical education literature on the benefits of using video 
feedback to inform motor skill development and improve student motivation (Potdevin et al., 
2018). 

While female cadets only make up about 24% of the students at the academy (PAO, 
2019), females account for about 40% of the low performers and failures in the military 
movement class. According to the end of year report (Goetz, 2020), there were 68 course failures 
in the academic year 2019-2020. Goetz found that female students accounted for 27 of the course 
failures and failed the course at twice the rate of the enrolled males. The lower performing 
female students struggle to score points on many of the physically demanding tasks in the course. 
Low performing students lack intrinsic motivation and self-confidence (Erturan & Hulva, 2019) 
and are therefore less likely to use their free time to work on their skills. Given the minimal time 
for classroom instruction and modeling (Coelho, 2019), instructors seek ways to boaster student 
participation and buy-in so that they benefit from the feedback provided by instructors 
(Chatoupis & Vagenas, 2018). 

This action research case study sought a more efficient method of providing female 
students individualized feedback with the goal of decreasing the physical skill gap between the 
males and females at the academy. The purpose of this study was to determine how video 
feedback effects skill performance in physical education classes and the perceived benefits of 
video feedback for the low performing females in an applied gymnastics class. The research 
questions guiding this study were: 

1. What are female cadets’ perceptions of video feedback for skill evaluation and skill 

improvement? 

2. How do female cadets perceive video as an additional method of feedback? 

3. How do female cadets perceive video feedback as a method to promote motivation for 

deliberate practice? 

4. How do female cadets’ perceptions of their skills change when video is added as an 

additional method of instructor feedback? 

Literature Review 

The two largest factors for inability to perform gross motor skills are a lack of physical 
abilities (Evans, 2013) or the lack of productive feedback (Roure et al., 2019). Research of 
Vanderhasselt et al. (2018) suggests that gender effects the prevalence of confidence and coping 
skills. Furthermore, current research suggests female students are less achievement motivated 
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compared to male students in physical education (So-Chen et al., 2016; Ulstad et al., 2019). 
Consequently, female students may respond by lowering their expectations for success in 
challenging tasks and avoiding the stressor (Yeung, 2011). Additionally, Yeung found that 
female students may be inclined to try harder to overcome the difficulties they encounter in the 
classroom. Outcomes from traditional gender stereotypes have contradictory findings. Berlin and 
Dargnies (2016) found female students are more apt to cope and avoid in the presence of 
stressors due to a fear of failure. They reported that female students displayed greater degree of 
effort in challenging tasks. Vanderhasselt et al. (2018) found that emotional based coping, 
avoidance of problems brought about by stressful situations, predicted higher levels of anxiety 
and depression. Research suggests that some females appear more apt to cope and avoid due to a 
fear of failure. 

A limiting factor to skill acquisition for students is the lack of quality feedback during 
instruction (Turner & West, 2013). Augmented feedback is a vital part to learning (Kangalgil & 
Ozgiil, 2018). According to Smith (2011), instructors must present augmented feedback to the 
learner as often and as soon as possible after an attempt to enhance the evaluation of movement 
and reinforce the memory representation. Providing clear and concise feedback will help 
students properly practice and develop the desired physical skills (Hatzipanagos & Warburton, 
2009). Efficiency will generate more time to spend on skill mastery (Smith, 2011). So-Chen et 
al. (2016) concluded instructors need to seek efficient teaching methods to achieve course goals 
and to provide their students with correct information to enhance learning, behavior, knowledge, 
and positive student attitudes. Erturan (2014) found the nature of teaching styles, type of 
feedback, time spent on the task, size of the class, and the nature of learning content are the most 
crucial factors that are related to teaching effectiveness. In addition, instructors need to foster an 
effective teacher-student interaction that provides a participatory environment for all students 
(Griffin et al., 2013). Overall, effective feedback should increase student motivation (Cecchini et 
al., 2019) while giving positive reinforcement of goals (Baghurst et al., 2015) and showing the 
correct and incorrect actions in the execution of a skill (Hattie & Timperley, 2007). Instructor 
feedback enhances student learning (Sharma et.al., 2016) through an evaluation of their 
contributions, discrepancies, and how to fix their errors (Berlin & Dargnies, 2016). 

Physical education teachers are in a unique position to capitalize on the use of video in 
the classroom (BenitezSantiago & Miltenberger, 2016). Potdevin et al. (2018) found video 
modeling and video feedback are effective tools to aid trainers. Boyer et al. (2009) examined the 
use of expert video modeling combined with video feedback of performance execution of three 
complex gymnastic skills. While the athletes were not able to achieve perfect performance 
during the study, the gymnasts all demonstrated increases in skill performance and improved 
skill retention after the intervention. Video analysis gives a degree of freedom and choice for the 
student’s feedback reception (Laughlin et al., 2019). Students can also incorporate select peer 
feedback to go with the review of video recordings (Potdevin et al., 2018) to review their own 
trials and self-assess, thus taking charge of their learning process (Laughlin et al., 2019). Video 
analysis for every lesson and every skill is not practical. However, video analysis can serve as an 
effective tool for supporting quality instruction and assessment of skills. 

Four theoretical frameworks that underpinned this research were (A) social learning 
theory (Bandura, 1977; Chng & Lund, 2018); (B) self-determination theory (Baghurst et al., 
2015; Drost et al., 2018; Ryan & Deci, 2000); (C) motor learning theory (Schmidt, 1975; Sharma 
et al., 2016); and (D) theory of feminism (Avci, 2016; Corey, 2009). 
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Methodology 

Action research design connects theory to practice and has a connection to school 
improvement (Mertler, 2017). This study used a mixed-methods approach utilizing data from 
instructor observations, pre/post-test questionnaires, semi-structured interviews, video 
recordings, and student self-assessments. Through extreme case sampling, the low performing 
female students (N=10) within the 8-week gymnastics course were provided video feedback on 
their graded attempts of three gymnastic skills. 

Setting and Participants 

The research site was a four-year military service academy. The school enrollment was 
4,400 with 20% women (PAO, 2019). The study occurred during the 8-week, 19-lesson military 
movement course that exposed cadets to a variety of basic movement skills, with 28 physical 
skills being taught and tested. Each 50-minute session consisted of roughly two skills per lesson 
with structured time for a master demonstration, teaching the skill progression, free-practice 
time, and testing (Coelho, 2019). The military movement class is a graduation requirement that 
all students must complete during their freshman year to obtain a commission in the United 
States Army. 

The student to teacher ratio in the class was 10:1. Each class had between 35-40 students. 
The average class consisted of about 30 males and 5-10 females. Extreme case sampling was 
used to select the lowest performing females in the class (those who actively displayed avoidance 
and poor coping techniques during skill practice time). Ten female students who met this 
criterion and were currently enrolled in the military movement course volunteered to participate 
in the study. There was no extra incentive to take part in the study beyond the potential benefits 
of the intervention. All participants returned signed copies of the informed consent prior to 
engaging in the study and were assigned unique IDs. The age range of the participants was 18-23 
with an average age of 19.30 (SD=1.40). The majority (66.7%) of the participants were 
Caucasian with the remaining having a diverse mix of ethnicities. No additional demographic 
information was collected. The participants also described their overall fitness level at a range 
from fair to excellent with a median of fair and had no limiting physical profiles that inhibited 
their ability to complete the requirements of the course. 

Intervention 

The study coincided with an eight-week quarter of the academic year to align with a class 
in session. The study took place over the course of six weeks during the additional instruction 
time outside of the regularly schedule class time. The program of instruction for the class 
included additional instruction periods scheduled throughout the quarter. The purpose of the 
additional instruction sessions was to provide lower performing students within the gymnastics 
course an opportunity to practice, review video feedback on their graded attempts of three 
gymnastic skills to identify performance deficiencies, and retest on their class skills. The 
procedures for the intervention were divided into three stages: pre-test, test, and post-test. 

The pre-test stage included all the activities the participants performed prior to 
participation in the intervention. The method of skill progression was in accordance with the 
instructor manual (Coelho, 2019). Participants also completed the pre-test questionnaire. The test 
stage began once the students had received all the lessons covering the selected skills and 
covered all the graded attempts during the intervention with the use of video feedback and 
instructor feedback to influence skill acquisition. The participants learned the performance cues 
of the selected skills during the class periods. The students were provided a master 
demonstration followed by progressions of each skill. After each graded attempt, the participants 
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were allowed three minutes of deliberate practice time to work on their deficiencies prior to 
taking their next graded attempt of the skill. Effective use of practice time during the study was 
defined as a participant using the practice time to actively work on any of the components of the 
assigned skill. Verbal cues on the points of performance that the student needed to fix to earn a 
higher score on the skill were provided by the instructor. The instructor also replayed video of 
the attempt with the student to identify performance deficiencies. Once the student felt 
comfortable with the skill, they were able to make one or two graded attempts with instructor 
feedback on each attempt. Once the attempts are made for all three skills, the final stage of the 
intervention, the post-test stage, occurred. At this point, the participants completed the post-test 
questionnaire and individually met with the lead researcher for a semi-structured interview 
concerning the process and the intervention. 
Data Collection 

This study used multiple methods of data collection including pre/post-test student 
questionnaires, instructor observations, semi-structured interviews, video recordings of graded 
attempts, and student self-assessments. All participants completed three attempts of three 
separate gymnastics skills: cartwheel, rope climb, and shelf mount. 
Pre-Post Test. The study used the earlier work of Standage et al. (2005) and their assessment of 
self-determination theory in school physical education for the pre/post-test questionnaire 
measures of autonomy, perceived competence, and motivation. Measures of autonomy included 
six items that corresponded to the participant’s level of familiarity and comfort with the selected 
skills. During the pre-test stage, prior to each attempt, students provided an estimate of their 
projected score on the attempt with measures of competence assessed using six items that were 
modified from the perceived competence subscale of the 18-item Intrinsic Motivation Inventory 
previously created by McAuley et al. (1989). Measures of motivation to learn were captured 
using 10 items, four items to assess levels of intrinsic motivation, four items to assess a lack of 
motivation, and two items to assess levels of extrinsic motivation. The questionnaire used a 5- 
point Likert scale (1= Strongly disagree to 5 = Strongly agree). Additionally, the distinct types of 
motivation were assessed using the Perceived Locus of Causality scale developed by Goudas et 
al. (1994). The nine items were divided into three subscales that examined intrinsic, extrinsic, 
and a lack of motivation for the military movement class. The scores from these three subscales 
were used as indicators for motivation. A high score in a subscale reflects a strong indication of 
agreement with the motivational methods. A low score in a subscale reflects a lack of agreement 
with the motivational method. 
Instructor Observations. Observations of the participant’s active use of their deliberate practice 
time was recorded on a researcher created data sheet. The number of practice attempts that each 
student took was noted on the same data sheet. Notes on whether the participant spent time 
viewing their video feedback again during the practice session was also captured on the data 
sheet. 
Semi-Structured Interviews. Semi-structured, one-on-one interviews were conducted with the 
participants before and after the intervention. Twelve open-ended questions were used to 
measure the benefits the cadets identified during the intervention. The interviews also assessed 
how giving video feedback impacted the instructor/cadet process for skill evaluation and 
improvement. 
Video Recordings. The instructor video recorded each graded attempt of the skills using standard 
video recording settings on the participant’s phone to inform the student about their quality of 
movement during the execution of the physical task. The instructor focused the feedback on the 
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visual depiction of the verbal cues of the skill being tested. The participant then viewed the 
recording of their attempt with the instructor to identify the key areas for skill improvement. 
Using the video recording, the instructor visually showed the student what elements of the skill 
were performed correctly, incorrectly, as well as gave the participant a grade for the attempt. 
Assessments. Cadets provided an estimate of the score they felt they would get when they 
performed the skill for a grade prior to each attempt execution. The students also provided a 
rating level to address their previous experience and comfort level with each task prior to the first 
attempt. The experience level was based on a 5-point Likert scale (1= J have never heard of the 
exercise to 5 = Iam an expert). The attempt score was also collected based on the performance 
scores of the participants. Data for each participant was collected at the end of each session for 
performance on the task. 
Data Analysis 

The quantitative data were collected from the questionnaires and assessments. 
Quantitative data were analyzed to illustrate the rate of skill acquisition and improvement as well 
as the level of motivation of participant. The scores were analyzed for each skill using repeated 
measures ANOVA to investigate changes in the mean over the course of three or more points in 
time. A Pearson’s correlation coefficient was used to measure the statistical relationships. 
Qualitative data was analyzed for themes through content analysis to reflect participants 
perceptions, feelings, attitudes, and opinions of the study’s intervention. 

Results 

On the pre-test, measures of autonomy corresponded to the participant’s level of 
familiarity and comfort with the selected skills. The participants indicated a median value of 
familiarity of slightly familiar with the cartwheel, very familiar with the vertical rope climb, and 
moderately familiar with the shelf. The participants indicated a median comfort level of neutral 
with the cartwheel, comfortable with the vertical rope climb, and uncomfortable with the shelf. 
The participants’ self-confidence and competence responses indicate a mixed level amongst the 
group. Most of the group (90%) felt they would be able to perform the skills well on their third 
graded attempt. However, the group was split concerning their satisfaction of their performance 
of the skills. Only 40% of the group agreed they would be satisfied with their performance while 
30% disagreed. The participants overwhelming believed the class would boost their confidence. 
Most of the group (90%) felt they would be confident on the skills after working on the tasks in 
class. When comparing their performance on the skills with their peers, only 30% of the group 
felt they would do well compared to their peers. The participants indicated an elevated level of 
support for intrinsic motivation. Nearly all participants (90%) agreed that they find “pleasure and 
satisfaction” in learning new things. Most of the group (90%) agreed they go to the class to 
“prove to themselves” they can complete the class. However, only 30% of the participants agreed 
that military movement class teaches “things that interest me.” The participants also indicated an 
elevated level of support for extrinsic motivation for the course where 60% of the group agreed 
the course would help them “better prepare for military career.” Additionally, only 40% of the 
group agreed that success in the class will make them “feel important.” The participants also 
indicated a low level of support for a lack of motivation for attendance and participation in the 
military movement course. The bulk of the group (80%) felt like they understood the purpose of 
military movement and why they needed to take the course. Likewise, 70% of the group did not 
feel like the class was a “waste of time.” However, 20% of the group agreed that they did not 
“understand what they are doing” in the class. Overall, the group indicated an elevated level of 
motivation to attend the class and saw the value and importance of taking the class. 
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The participants were asked to predict their attempts just prior to making each of their 
three video recorded attempts. The predicted scores were based on their previous exposure to the 
skill during the class instruction and deliberate practice sessions. The mean predicted score for 
all video recorded attempts of the cartwheel was 2.37 (SD=1.61). The mean predicted score on 
the vertical rope climb was 3.67 (SD=1.69). The mean predicted score on the shelf was 3.70 
(SD=1.90). After the students provided their predicted performance scores, they completed one 
graded attempt of the skill. The graded attempt was recorded using the student’s video recording 
application on their cell phone. The mean graded score for all video recorded attempts of the 
cartwheel was 1.47 (SD=1.53). The mean graded score on the vertical rope climb was 3.13 
(SD=2.16). The mean graded score on the shelf was 3.33 (SD=2.14). 

Using a repeated measures ANOVA, the findings indicate there was a statistically 
significant increases in scores within the group for the cartwheel and the shelf. Video feedback 
had a statistically significant effect on the performance of the cartwheel, F(2, 18) = 21.50, p< 
.001. Additionally, the participants significantly increased their performance on the shelf, F(2, 
18) = 3.58, p = .049. Contrary to the cartwheel and shelf, the participants did not significantly 
increase their performance on the vertical rope climb, F(2, 18) = 2.62, p = .10. However, the 
group did increase their mean score on the vertical rope climb from their first attempt, M= 2.90, 
to their final attempt, M= 4.10. Moreover, the students made significant adjustments to their 
predicted scores on the shelf between attempts, F(3, 27) = 4.23, p = .014. The group did not 
make significant adjustments to their predicted scores on the cartwheel, F(3, 27) = 2.57, p = .075, 
or the rope, F(3, 27) = 2.28, p = .10. While the changes in the group’s predicted outcomes were 
not statistically different for the cartwheel or rope, the difference between the predicted and 
actual scores for the individuals within the groups improved significantly for the cartwheel. 

The participants overestimated their abilities on the cartwheel by an average of 1.0 point 
on their first attempt and underestimated themselves by an average of 0.20 points on their final 
attempt. The improvement in their accuracy of predictions on the cartwheel was significant, F(2, 
18) = 14.48, p < .001. The participants had overestimated their abilities on the vertical rope 
climb by an average of 0.40 points and underestimated their skills by an average of 0.40 points 
on their final climb. The group did not significantly improve the accuracy of their predictions on 
the vertical rope climb, F(2, 18) = 3.23, p = .063. While the group did make significant changes 
to their predictions on their performance of the shelf, they did not significantly improve the 
accuracy of their predictions, F(2, 18) = 2.16, p = .14. The group went from overestimating their 
skills on the shelf by an average of 0.20 points to underestimating their scores by 0.50 points. 

Overall, the group saw their mean scores improved in each skill. The group had a mean 
score increase of 1.50 (SD=.71) on the cartwheel. The group had a mean score increase of 1.20 
(SD=1.69) on the vertical rope. Additionally, the group had a mean score increase of 1.40 
(SD=.97) on the shelf (see Figure 1). 

The Pearson’s r correlation analysis for the cartwheel revealed limited correlation 
between the number of practices and the final score, =0.37, as well as the relationship between 
practices and score improvement, r=0.52. However, for the vertical rope climb there was a strong 
positive correlation found between practices and score improvement, 7=0.88, p < .001. There 
was minimal negative correlation found for the shelf between practices and score improvement, 
r=-0.19, and nearly no correlation between practices and their final score, 7=-.05. 
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Figure 1 
Average Scores for the Predicted and Actual Attempts 
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Data analysis revealed skill improvement in the participants. Overall, all the participants 
improved their task score on at least one event. Furthermore, half of the participants increased 
their task score on all three events. All the participants identified the video feedback as helpful 
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for improving at least one of their skills. Within the group, 70% of students reported an increase 
in perceived competence, 90% reported an increase in autonomy, and 80% of the students 
reported an increase in motivation. Overall, 90% of the participants recommended the future use 
of the video feedback in the performance of each of the performance skills. 

Major themes that emerged during the qualitative data analysis included clarity of the 
feedback, changes in self-awareness, changes in deliberate practice, and changes in performance. 
The participants identified many benefits from the use of video feedback to inform their skill 
development. Video helped provide greater clarity for the knowledge of performance cues for the 
student as well as a better understanding of their deficient components of the skill. The video 
feedback was also shown to increase the accuracy of their predicted performance. Another 
significant finding from the study was the increase in self-regulation and autonomy that also lead 
to an increase in the use of deliberate practice time. The participants also experienced an increase 
in their motivation to learn while identifying support for both intrinsic and extrinsic motivational 
levels for the class. The study also found video feedback was beneficial for the participants’ 
increased self-awareness and the participants preferred video feedback as their method of 
instructor feedback in physical education classes. 

Implications for Practice 

This study aimed to improve the feedback process used in the military movement class at the 
service academy by generating data concerning the efficiencies and effectiveness of video 
feedback. The findings from this study support video recordings also offer multiple benefits that 
could assist physical education instructors as well as students seeking a more efficient and 
effective feedback process. Instructors are encouraged to utilize video feedback throughout the 
physical education curriculum to enhance instruction for lower performers within the program. 
The findings of this study also suggest video feedback is an effective method of augmented 
feedback for students struggling with novel physical motor skill progressions. These findings 
demonstrated that students perceived video feedback as an effective method for enhancing skill 
improvement in their gymnastics class. Video feedback helped the study participants provide 
greater clarity for the knowledge of performance cues as well as providing them a better 
understanding of their deficient components of the skill. Additionally, these findings indicated 
video feedback could be used to improve motivation, deliberate practice, competence, and 
autonomy. The utilization of video feedback was shown to improve participants task scores as 
well as increase the accuracy of their predicted performances. Finally, the participants found 
video feedback to be their preferred method of feedback for learning physical skills. They found 
video feedback to be more beneficial than instructor verbal cues on all performed skills. The 
augmented feedback provided them a clear understanding of their current ability level as well as 
an improved understanding of the areas required to improve. Suggesting that the ability to 
visualize performance cues for the students may also result in faster motor skill acquisition. 

Another suggestion for the implementation of video feedback is to incorporate peer feedback 
into the process. Allowing a partner feedback loop reduces the requirement for the instructor to 
serve as the lone source of feedback in the learning process. A peer could video record a practice 
attempt and then review the footage with the student. Reviewing the video should help students 
gain awareness of the key components of the skill while seeking methods for how to improve 
upon their deficiencies. In addition to helping reduce the time requirements of reviewing video 
during the class period, students could have their attempts recorded and then review the footage 
after class on their own time. 
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Limitations 

The study relied on self-reported ratings for perception of feedback effectiveness. The 
study also included only the lowest performing female members of the class. Higher performing 
students may not see as much benefit from video feedback regarding their skill development or 
as a source of motivation. Another limitation of the study was a small sample size of ten 
participants which decreased statistical power of the study and is not representative of the student 
population at large. However, the small sample size was selected due to the population of 
available participants for the study. Finally, the study examined a unique population and unique 
set of skills. While the fitness levels of the selected participants may be below average of their 
peers at the academy, they have an average level of fitness when compared to their peers at other 
universities. The selected skills are unique to the requirements of the military academy and are 
not a common skill found in many physical education courses. 
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Graduate Instructors’ Technostress of Engaging in Emergency Online Teaching During the 
COVID-19 Pandemic 


Abstract 


This study investigates graduate instructors' technostress before and during the COVID-19 
pandemic, and the relationship between instructors’ technostress with their TPACK 
competencies and institutional-colleague support, respectively. Results show a significant 
increase in instructors’ technostress level during the COVID-19 online teaching period (¢ (27) = - 
5.74, p<.01). Instructors’ TPACK competencies were significantly and negatively correlated 
with their technostress, while the institutional-colleague support was not significantly associated 
with their technostress. The findings suggested that institutions may consider providing 
pedagogy courses, including TPACK competencies, to help graduate instructors decrease 
technostress. 


Keywords: technostress, online teaching, TPACK, higher education 
Introduction 


In early 2020, most educational institutions in the United States rapidly transitioned from 
face-to-face instruction to online teaching due to the COVID-19 pandemic. According to Hodges 
et al. (2020), while this method is currently called “online teaching,” it is not planned online 
teaching but rather a crisis-promoted emergency remote teaching, which differs from online 
teaching before the pandemic. This teaching method is new to most instructors, and the use of 
technology has become a necessity rather than an option (Ozgiir, 2020; Panisoara et al., 2020), 
which may have led to stress deriving from technology use during online teaching (Joo et al., 
2016). Technostress, first introduced by Brod (1984), is defined as “a modern adaptation disorder 
resulting from the inability to use current computer technologies effectively” (Ozgiir, 2020, p. 1). 
While technostress research has been conducted extensively across different workplace settings, 
studies investigating educators’ technostress are still rare (Coklar et al., 2017). Among the few 
studies, Boyer-Davis (2020) discovered that college faculty suffered significantly more 
technostress during the pandemic than before the pandemic. Estrada-Mufioz et al. (2020) also 
found that during the pandemic, instructors’ stress and anxiety associated with the application of 
educational technology grew exponentially over time. These implied that more studies are 
needed to understand educators’ technostress to better support them in navigating the emergency 
online teaching. 

Across the different groups of educators, graduate instructors received very little concern 
in educational research. But in reality, graduate instructors have become a significant teaching 
force for supporting undergraduate education in the United States (Douglas et al., 2016), and the 
preparedness of graduate instructors could directly impact the quality of education (Fong et al., 
2019). Despite its significance, the support and training provided to them were quite limited 
(Luft et al., 2004), and few studies have been conducted to evaluate the impact of this support 
provided (Fong et al., 2019; Wyse, 2010). To better understand graduate instructors’ teaching 
conditions in the United States, this study investigates their technostress levels before and during 
the COVID-19 pandemic and examines the factors that led to their technostress and ways to cope 
with it. 
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Literature Review 


Numerous scholars have investigated factors influencing faculty’s technostress in 
teaching. Previous literature has noted that personal and environmental factors are major ones 
resulted in instructors’ technostress (Matthews et al., 2004; Ozgiir, 2020). 


Personal Factors 

In a study, Fuglseth and Sorebo (2014) discovered that the instructors’ low-level 
individual ability (e.g., technological literacy, mental competencies) was an essential reason for 
their technostress. Some other personal factors, such as instructors’ negative beliefs and 
resistance to applying computer technology into their teaching, were also identified by 
researchers (Harahap & Effiyanti, 2015). Likewise, a recent study by Zeeshan et al. (2020) 
examined the technostress of university faculty due to their lack of pandemic preparedness and 
sudden transition to online teaching during the pandemic in Pakistan. They discovered that in 
addition to technical issues such as internet connections, power disruptions, camera and voice 
quality, instructors’ personal factors (including fear of using technology in class) were resulted in 
their technostress during the pandemic. Panisoara et al. (2020) also claimed that instructors’ 
intrinsic motivation could effectively reduce their technostress. 

Among a number of personal factors, Joo et al. (2016) proposed that teachers’ 
competencies to teach with technology (developed from the Technological Pedagogical Content 
Knowledge - TPACK) are highly related to their technostress in teaching. TPACK is a 
framework that introduces the relationships between three basic components of knowledge 
(technology, pedagogy, and content) (Koehler & Mishra, 2008, 2006). Meanwhile, at the 
intersection of the basic knowledge are the four overlapping domains elaborate on integrating the 
three circles (Mishra & Koehler, 2006). In total, seven components are included in the TRACK 
framework (see Figure 1). Briefly, Technology Knowledge (TK) refers to the knowledge about 
various technologies (Schmidt et al., 2009), and Content Knowledge (CK) is the “knowledge 
about actual subject matter that is to be learned or taught” (Mishra & Koehler, 2006, p.1026). 
Pedagogical Knowledge (PK) refers to the methods and processes of teaching (Schmidt et al., 
2009), and Pedagogical Content Knowledge (PCK) is the content knowledge that deals with the 
teaching process (Shulman, 1986). Technological Content Knowledge (TCK) 1s related to the 
knowledge of how technology can create new representations of specific content (Schmidt et al., 
2009), and Technological Pedagogical Knowledge (TPK) refers to the knowledge of how various 
technologies can be used in teaching. Technological Pedagogical Content Knowledge (TPCK) 
refers to the knowledge required by teachers for integrating technology into their teaching in any 
content area (Schmidt et al., 2009). Many studies have been conducted to apply the TPACK 
framework into understanding teachers’ knowledge of integrating technology into teaching (e.g., 
Archambault & Crippen, 2009; Hughes et al., 2020). However, since our current study focused 
on understanding the influence of technology-related self-efficacy for online teaching during the 
pandemic, only TK, TPK, and TCK were used to examine graduate instructors’ technostress. 


31 


Figure 1. 
The Components of the TPACK Framework (graphic from http://tpack.org/) 
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Environmental Factors 

Previous studies (e.g., Fuglseth & Sorebo, 2014; Ragu-Nathan et al., 2008; Salanova et 
al., 2013) found a lack of environmental support, such as technical support, information- and 
communication-technology use facilitators, can result in technostress. Additionally, Al-Fudail et 
al. (2008) noted that the lack of fit between teachers and the technological environment, 
specifically, the unbalance between demands of the technological environment and teachers’ 
abilities, as well as teachers’ needs and supplies, cause instructors’ technostress. 

Joo et al. (2016) proposed that school support is an external factor, and the external 
regulation could enhance instructors’ stress to a large extent (Panisoara et al., 2020). Panisoara et 
al. (2020) also emphasized that the school context has a significant impact on teachers’ 
technostress. Vladut and Kallay’s (2010) study found that increased school demand for 
technology use creates technostress among instructors. Dong et al. (2020) also identified 
administration support that denotes infrastructure and technical assistance and collegial support 
from colleagues as critical environmental factors related to teachers’ use of technology. 

In summary, the above research findings suggest that the major reasons for teachers’ 
technostress include teachers’ personal and environmental factors. Dong et al. (2020) framed a 
structural model showing the relationships among teachers’ technostress, TRACK, computer 
self-efficacy, administration support, and collegial support. They found that TPACK can help 
teachers deal with psychological stress caused by technology use and suggest strengthening 
teachers’ TPACK skills through school support and increasing teachers’ computer self-efficacy. 
Ozgiir (2020) also noted a negative relationship between teachers’ technostress levels and school 
support and TPACK competencies. TPACK negatively affects teachers’ technostress, and 
teachers’ technological integration competencies could significantly decrease teachers’ 
technostress levels caused by ICT use during the education-training process. 

Overall, limited studies have been conducted to investigate technostress among graduate 
instructors, the associations between their personal factors, mainly TPACK competencies and 


32 


their technostress, and the relationship between the support they have received from institutions 
and colleagues with their technostress; not to mention their emergency online teaching during the 
pandemic (Estrada-Mufioz et al., 2020). This study will contribute to the current literature and 
investigate graduate instructors’ technostress during the pandemic. 


Method 
The Research Questions 


1. Is graduate instructors’ technostress significantly different before and during the COVID- 
19 pandemic when emergency online teaching began? 

2. Are there any associations between instructors’ TPACK competencies and their 
technostress regarding emergency online teaching? 

3. Are there any associations between institutional-colleague support and instructors’ 
technostress regarding emergency online teaching? 


Data Collection and Participants 

An online survey (see Appendix) with validated instrument items was used to answer the 
proposed research questions. Participants were asked to rate each item on a five-point Likert 
scale, ranging from one (strongly disagree) to five (strongly agree). First, participants were asked 
to rate their stress levels before and after classes were conducted in the online setting. A 7-item 
scale adopted from Panisoara et al. (2020) was used to measure technostress. An 8-item scale 
from Chai et al. (2011) was used to measure instructors’ TK and TCK, while 4-item scale each 
from Panisoara et al. (2020) and Valtonen et al. (2017) were used to measure instructors’ TPK 
and TPCK, respectively. Participants’ institutional-colleague support was measured by a 6-item 
scale from Dong et al. (2019). The last section of the survey was about the demographic 
information of the participants. 

The online survey was distributed to graduate instructors in a public university in the U.S. 
in February 2021, and 31 responses were collected. After excluding the invalid responses, 28 
participants were included in this study. Descriptive demographics of the students who 
completed the survey are demonstrated in Table 1. 


Table 1. 
Participant Demographics Information 
Value n % 
Demographics 
Gender Male 8 28.6% 
Female 16 57.1% 
non-binary/not to disclose 4 14.3% 
School/Department College of Education 21 75.0% 
School of Engineering 4 14.3% 
College of Natural Sciences 3 10.7% 
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Data Analysis 

Survey responses were analyzed using R. First, mean (M) and standard deviation (SD) 
were calculated to get an overall understanding of graduate instructors’ technostress levels, their 
TPACK competencies, and institutional-college support. Then, paired-sample t-tests were used 
to compare their technostress before and during the pandemic. Pearson’s correlation analyses 
were conducted to investigate relationships between instructors’ technostress and their TRACK 
competencies, and between technostress and their institutional-colleague support. 


Results 


Instructors’ Technostress Levels Before and During the Pandemic (RQ1) 

As shown in Table 2, descriptive statistics demonstrate that instructors’ technostress 
levels before COVID-19 pandemic (M = 2.71, SD = 0.81) were lower than when courses were 
transferred online (M = 3.43, SD = 0.99). The paired samples t-test result indicates that 
instructors’ technostress levels were significantly higher after switching to online teaching (t (27) 
= -5.74, p < .01). Its effect size is 0.79 (95%CI [0.73, 0.85]), a notably large effect size (Cohen, 
1992). This result suggests that graduate instructors felt much more technostress when the 
courses were transferred from in-person to online format. 


Table 2. 
Paired Samples t-test of Graduate Instructors’ Technostress Before and During the COVID-19 
Pandemic (n = 28) 


Mean SD t 
Pair Before the pandemic 2.71 0.81 -5.74** 
During the pandemic 3.43 0.99 


** D<.01 


The Association Between Instructors’ TPACK Competencies and Their Technostress 
Levels (RQ2) 

Descriptive statistics show that instructors had different competency levels in each 
TPACK dimension (see Table 3), with TPK, TPCK, TK, and TCK in descending order. A 
Pearson’s correlation between instructors’ TPACK and their technostress about emergency 
online teaching (M = 2.75, SD = 0.98) was also conducted. As shown in Table 4, graduate 
instructors’ technostress about emergency online teaching negatively correlated with their TK, 
TPK, TCK, TPCK. The effect sizes (0.49, 0.37, 0.41, 0.4) indicate that a moderate amount of the 
variability in instructors’ technostress was explained by their TRACK competencies. These 
findings suggested that when graduate instructors have a higher level of TRACK competency, 
their technostress could be lower. 
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Table 3. 
Descriptive Statistics of Graduate Instructors’ TPACK Competency Level 


Dimensions Mean SD 

Technological Knowledge (TK) 3.90 0.93 
Technological Pedagogical Knowledge (TPK) 4.07 0.72 
Technological Content Knowledge (TCK) 3.50 0.82 


Technological Pedagogical Content Knowledge (TPCK) 3.96 0.81 


Table 4. 

Pearson’s Correlation for Graduate Instructors’ TPACK and Technostress 
Variable Technostress 
Technological Knowledge (TK) -0.70** 
Technological Pedagogical Knowledge (TPK) -0.61** 
Technological Content Knowledge (TCK) -0.64** 
Technological Pedagogical Content Knowledge 0.63** 
(TPCK) 

** n<.01 


The Associations Between Institutional-colleague Support and Instructors’ Technostress 
(RQ3) 

Regarding environmental support, instructors received nearly equal support from 
institution (M = 3.81 SD = 0.59) and colleagues (M = 3.82, SD = 0.98). 22 (78.6%) participants 
“agreed” or “strongly agreed” that their college or school provides enough infrastructure and 
resources for them to do well with online teaching, while 15 (53.6%) participants “agreed” or 
“strongly agreed” that their college or school provides clear guidelines about online teaching. 
Regarding support from colleagues, 21 (75.0%) “agreed” or “strongly agreed” that they and 
colleagues made a connected effort to integrate technology into online teaching; 17 (60.7%) 
agreed or “strongly agreed” that they received encouragement from colleagues when they 
encountered difficulties integrating technology into their online teaching. 

However, Pearson’s correlation analysis indicated that there were no significant 
correlations between the institutional (7 =0.12, p>.05) and colleague support (7 =0.253, p>.05) 
with instructors’ technostress about emergency online teaching. 
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Discussion 


Technostress Levels and TPACK Competencies 

Our study analyzed graduate instructors’ technostress levels before and during the 
pandemic. It discovered that graduate instructors’ technostress levels significantly increased 
when the courses were transferred from an in-person to an online format during the pandemic. In 
previous studies, the demand for a technological environment in the teaching context was an 
important factor that triggered instructors’ technostress (AI-Fudail & Mellar, 2008; Panisoara et 
al., 2020). In this study, the transition from a familiar teaching format (i.e., in-person teaching) to 
online teaching during the COVID-19 pandemic was an external regulation and requirement for 
the graduate instructors, and they had little time to prepare themselves for this change in the 
teaching environment. Hence, it is not surprising that the graduate instructors experienced higher 
technostress levels after switching to online teaching. 

Additionally, their technostress levels were significantly negatively correlated with their 
TPACK competency levels. As Ozgiir (2020) pointed out, instructors’ technostress levels are 
determined by their abilities and competencies in using technology, and teachers’ TPACK 
competencies greatly decrease their technostress levels. The significant negative correlations 
between graduate instructors’ technostress levels and their TPACK competency levels found in 
the current study align with the previous literature showing that teachers’ TPACK competencies 
could contribute to decreasing their technostress levels. This finding indicates that graduate 
instructors’ TPACK competencies play an important role in helping them cope with technostress, 
a result similar to those in previous studies (e.g., Al-Fudal & Mellar, 2008; Joo et al., 2016). 
Among the TPACK dimensions, TK was found to be most negatively correlated with graduate 
instructors’ technostress (see Table 4); thus, improving graduate instructors’ technological 
literacy could help decrease their technostress. Also, because TK was not content- or subject- 
related, a general technology training workshop on integrating technology into teaching could 
benefit graduate instructors. 


Institutional-colleague Support and Instructors’ Technostress 

Our findings suggested that only half of the graduate instructors agreed or strongly 
agreed that their college or school provides clear guidelines for online teaching, indicating that 
they did not get enough support from their institutions. The graduate instructors who participated 
in our study also pointed out the importance of getting support from peers, suggesting 
collaborative work among graduate instructors on integrating technology into teaching. 
However, our findings indicated that the support graduate instructors received from institutions 
and colleagues did not significantly correlate with their technostress. The significant correlations 
between graduate instructors’ TPACK competencies and their technostress, and no significant 
correlations between the supports graduate instructors received from the environment and their 
technostress, suggested that the personal factors played a more influential role in determining 
graduate instructors’ technostress. Thus, a more effective and practical way to reduce graduate 
instructors’ technostress could be to improve their TPACK competencies. 

Current literature has suggested that the pedagogy course can consistently increase 
graduate instructors’ confidence and attitudes toward teaching (Fong et al., 2019). The findings 
of our study could contribute to the current literature and expand their suggestion that the 
TPACK competencies should be an essential component in the pedagogy course for graduate 
instructors. Dong et al. (2020) proposed that institutions should create more opportunities for 
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teachers to accumulate hands-on experience and provide timely feedback to improve teachers’ 
perceived knowledge and abilities and reduce their technostress (Xie et al., 2017). They also 
suggested that mutual collaboration among colleagues effectively enriches instructors’ 
development of TPACK better than providing sufficient infrastructure and technical training. 
Based on our findings, we also indicated that for the graduate instructors, the collaboration-based 
learning in a pedagogy course could also benefit them to gain knowledge and reduce 
technostress. 


Conclusion 


Graduate instructors provide instruction for many university classes. They have become 
an increasingly indispensable part of the teaching group; however, the current literature has not 
thoroughly investigated these young instructors’ technostress. Our findings suggest that 
developing graduate instructors’ TPACK competencies is a viable way to reduce technostress 
caused by the rapid move to online teaching. This study has practical implications for institutions 
supporting graduate instructors, in particular, providing courses for graduate instructors to 
improve their teaching ability and TPACK competencies. This study has limitations. Because it 
is small in scale and the participants were recruited from one university, our conclusions need to 
be further tested in larger-scale studies and other higher education contexts. 
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Appendix 
Survey 
1. Please rate your stress level before Spring break 2020, when classes were conducted in a 
face-to-face setting. (Extremely high/High/Medium/Low/Not at all) 
2; Please rate your stress level after Spring break 2020, when classes were conducted in an 
online setting. (Extremely high/High/Medium/Low/Not at all) 
2 Please respond to the items below based on your feelings about the emergency online 


teaching during COVID-19 pandemic. (strongly disagree/disagree/neutral/agree/strongly agree) 
a. I feel stressed while adapting myself to online teaching. 
b. I find it difficult to use digital technology effectively for online teaching due to my 
limited time availability 
c. I feel stressed by the high technical requirements that are necessary for online teaching. 
d. I find it difficult, with my current skills, to constantly improve the act of online teaching 
e. I feel that online teaching complicates my teaching activity. 
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f. Itis hard for me to concentrate on teaching with the use of different digital tools in the 
online setting. 
g. I hesitate to incorporate other digital tools during online teaching for fear of making 


mistakes. 

h. Iam worried that online teaching will compromise student learning quality. 
4. Please answer the following questions about your technological knowledge for teaching 
online. 


a. I know how to solve my own technical problems for online teaching. 
b. I can learn technology easily to serve the purpose for online teaching. 
c. I keep informed about new digital technologies about online teaching. 
d. I have the technical skills to use technology effectively for online teaching. 
5. Please answer the following questions about your technological pedagogical knowledge 
for teaching online. 
a. Ican help my students use online learning environments (e.g., Canvas, Google 
applications) effectively. 
b. I can design lessons/courses that enhance the online teaching approaches. 
c. Ican use appropriate digital conferencing technologies (e.g., Zoom, Skype, Google Meet, 
WebEx etc.), which allow me to communicate and interact synchronously with other 
colleagues or students. 
d. I can use online tools to assess students’ knowledge. 
6. Please answer the following questions about your technological content knowledge for 
teaching online. 
a. I know digital technologies which I can use to illustrate difficult contents in my online 
course. 
b. I know websites with content materials for studying my online course. 
c. I know how to decide on the technologies that will enable students to learn the contents 
of my online course in a meaningful way. 
d. I can use the software that are created specifically for my online course. (e.g., e- 
dictionary/corpus for language; Geometric sketchpad for Math; Data loggers for Science). 
7. Please answer the following questions about your technological pedagogical content 
knowledge for teaching online. 
a. In teaching a specific online course, I know how to use digital technology as a tool for 
students to plan their own learning. 
b. In teaching a specific online course, I know how to use digital technology as a tool for 
students’ collaborative learning. 
c. In teaching a specific online course, I know how to use digital technology as a tool for 
students’ creative thinking. 
d. In teaching a specific online course, I know how to use digital technology as a tool for 
students’ critical thinking. 
8. Please answer the following questions about the ways you have been supported for the 
emergency online teaching during COVID-19 pandemic. 
a. My college/school provides clear guidelines about online teaching, so that I know how to 
guide student learning 
b. My college/school provides sufficient training so that I know how to implement online 
teaching 
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c. The requirement and timetable for implementing online teaching is reasonable in my 
college/school so that I can at my own pace 
d. My college/school provides enough infrastructure and resources so that I could do well 
on online teaching 
e. I get encouragement from my colleagues when I encountered difficulties in integrating 
technology in online teaching 
f. My colleagues share useful resources with me for online teaching 
g. My colleagues share their experience with me about integrating technology into online 
teaching. 
h. My colleagues and I made a concerted effort to integrating technology into online 
teaching 

9. What is your gender? 

10. What college/school are you in? 

11. How long have you been teaching in higher education? 
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Students’ Exam Taking Behaviors in Live and Online Exams 
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Abstract 


A Doctor of Pharmacy program is using a specialized examination software, ExamSoft® 
to addesss accreditation requirements. The primary objective of this study was to use ExamSoft 
snapshots log to provide empirical evidence of exam taking behaviors and the potential impact of 
the pandemic context on exam taker behaviors across different examination contexts. Significant 
behavioral differences were found between high and low achieving students as well as between 
face-to-face and online live monitored exams. 


Motivation and Objective of the Study 


The use of multiple-choices exams combined with the introduction of computer-testing 
platforms increased educational researchers’ interest in test-taking behaviors, changes in these 
behaviors, and their relationships with the assessment activities (Olev & Must, 2013; Plake & 
Wise, 1988; Tamowsky, 1977). As interest in test-taking behaviors increased, researchers started 
to focus on more specific impact areas such as the potential impact of test-taking behaviors on 
students’ cognitive test performance (e.g., Heinonen et al., 2011), the role of self-efficacy in 
mediating the impact of test-taking behaviors on test performance (e.g., Birenbaum & Alhia, 
2013), or the association between test-taking behaviors and test performance for students with 
special needs (e.g., Pohl et al., 2016). 


Pharmacy programs are required by an accreditation agency to provide specific data to 
support students’ progress (ACPE, n.d.). To address these requirements, a PharmD program 
began using a specialized examination software, ExamSoft®, that allowed instructors to link 
learning objectives and outcomes to assessment items. ExamSoft was integrated with the 
learning management system and provided valuable statistical data used to monitor the quality of 
individual exams items and the whole exam. ExamSoft includes security features such as device 
lock during the exam, item randomization, and snapshots that record all choices students make 
during an exam. 

The primary objective of this study was to analyze students’ snapshots and provide 
empirical evidence of exam taking behaviors, their potential relationship to the exam 
performance, and the potential impact of the pandemic context on exam taker behaviors. 


Context of the Study 


At the beginning of Fall 2020 the university set up pandemic restrictions that allowed 
both a mixed-attendance and a fully synchronous online attendance option. Both students and the 
instructor had the option to use either the face-to-face or online format of the course. For live 
attendance and examination options, the classroom seating capacity was limited and face masks 
were required. 
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To address the needs of fully online students, ExamSoft Exam Integrity functionality 
offered strong security options including offline video, audio recording, and flagging of potential 
breaches of integrity. For the target pharmaceutical course, in the first half of the semester, when 
COVID 19 pandemic restrictions were more relaxed the course was live and additional 
classrooms accommodated a live first exam for the entire class. For the second part of the course, 
the conditions worsened and the instructor decided to teach and administer the second 
examination fully online for the entire class. This situation offered a unique opportunity to 
analyze the exam taking behaviors of the same group of students both in the live and online 
context and analyze the potential impact of the pandemic-related conditions on the examination 
process. 


Research Focus and Methodology 
Research Design 


We used a sequential mixed-methods design that built on a qualitative analysis of the raw 
ExamSoft data to identify significant examination behaviors followed by a comparative 
qualitative analysis of the identified significant exam-taker behaviors. The dependent variables 
used in this study were behavioral patterns, assessment item difficulty, and the maximum 
number of choices an exam taker made for each assessment item throughout the examination 
period. The independent variables were grade level (high, within top 20% of the exam grades or 
low, within the bottom 20% of the grades), nature of the exam (face-to-face or online with digital 
exam monitoring), and the correctness of the final answer (correct or wrong). 


Data Collection 


ExamSoft records the entire activity of the exam taker during the exam in what is known 
as “Exam Taker Snapshots”. This detailed information was intended to serve as a means to 
provide support for the instructor when checking and identifying potential attempts of cheating. 
While this “policing” tool is considered a needed and useful security feature, the Exam Taker 
Snapshots (or simply ExamSoft snapshots) can also provide support for tracking students’ 
observed exam-taking behaviors (Cernusca & Friesner, 2019). That is, for multiple-choice 
assessment items ExamSoft snapshots include all choices students made (e.g., view an 
assessment item, select a specific answer, change a previously selected answer) and the time- 
stamp associated with each choice. 

The ExamSoft snapshot logs are available through the software interface (Figure 1) and 
allow for basic sorting of the information by assessment item or time stamp. However, the 
ExamSoft snapshot logs can also be downloaded as an Excel spreadsheet and used for further 
more detailed analyses. 
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Figure 1 
Sample Snapshot Viewer in ExamSoft 


Snapshot Viewer 


Previous 1 2 Next Show: 50 | 250 | 500 | 1000 | 2500 


item # Snapshot # Item Type Time Stamp Trigger Response 

1. Choice 5:05:30 PM Answered Choices): E 
Choice 5:15:14 PM Navigation Choice(s): C 
Choice 5:12:30 PM Answered Choices): E 
Choice 5:12:16 PM Answered 
Choice 5:13:30 PM Answered 
Choice 5:04:12 PM Exam Start 
Choice 5:24:30 PM Answered Choice(s): C 
Choice 5:06:19 PM Answered 
Choice 5:22:30 PM Answered Choice(s): C 
Choice 5:17:30 PM Answered Choices): B 
Choice 5:23:30 PM Answered 
Choice 5:09:30 PM Answered 


Question numbering in the snapshot reflected the default organization of the exam even 
when questions were randomized. The spreadsheet format of the snapshots allowed for the 
organization of choices by question number and time stamp to reflect the exam taker behaviors 
for each question. In addition to the snapshot viewer the item analysis output from ExamSoft 
provided the correct answer and assessment item difficulty (Figure 2) 


Figure 2 
ExamSoft Item Analysis Output Sample 


Exam Takers = 70 KR20 = 0.84 Stdev = 14.93 Mean = 70.11 (77.90%) Median = 72.00 


_ 


a 087)  100.00% 66.67% 
 ) 
Question ID / Rev: 21562/ 1 : : % Selected 87.14 2.86 


5 = = Z = Point Biserial (rpb) 0.41 -0.25 -0.29 
Disc. Index 0.33 -0.10 -0.10 


Upper 27% i 1.00 0.00 0.00 
- - - - - Lower 27% 0.67 0.10 0.10 
Q: Based on th NOT likely to be caused by the drug? 


A: biurred vision 
=> * B: drooling 

C: constipation 

D: dry eyes 

E: urinary retention 


Participant Selection 
The snapshot of seven high performers that were placed in the first 20% of the class and 


seven low performers placed in the lower 20% of the class from both live and online exams were 
selected for this research study. 
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This number of participants was considered optimal considering that each exam used for 
this study had between 35 and 50 assessment items and for each assessment item the ExamSoft 
snapshot log recorded between two and 10 behavioral actions. 


Data Analysis 


Data analysis involved a qualitative phase that helped identify significant behavioral 
patterns and a quantitative phase where identified behavioral patterns quantified and used for 


statistical analysis using SPSS® (https://www.ibm.com/products/spss-statistics) statistical 


software. 
Qualitative Analysis Phase 


The structure of the ExamSoft Snapshots allowed for a qualitative analysis of the 
observed exam taking behaviors followed by grouping of these behaviors into specific behavioral 
patterns. As a first step in the qualitative analysis the snapshot Excel output was organized by 
assessment item and then by the choice (recorded exam taker action). The first choice for each 
assessment item was then color coded and a column with the correct answer for each item was 
added to help analyze the sequence of the choices made by the exam taker (see Figure 3). 


Figure 3 
ExamSoft Snapshots Organized by Item and Choice 


\Item# Snapshot# lItemType TimeStamp _ Trigger Response correct_answer 
| 3 1 Choice 4:04:43 PM Answered A 
2 Choice 4:14:16 PM Answered Choice(s):A 
3 Choice 4:33:04 PM Final Choice(s): A 
1 Choice 4:09:16 PM Answered Choice(s):A 
2 Choice 4:33:04 PM Final Choice(s): A 
1 Choice 4:05:11 PM Answered 

2 Choice 4:14:36 PM Answered Choice(s):A 
3 Choice 4:33:04 PM Final Choice(s):A 
1 Choice 4:07:18 PM Answered Choice(s):C 
2 Choice 4:33:04 PM Final Choicef{s): C 
1 Choice 4:11:31 PM Answered 

2 Choice 4:29:29 PM Answered Choice(s):D 
3 Choice 4:33:04 PM Final Choice(s): D 
1 Choice 4:03:43 PM Answered Choice(s):A 
2 Choice 4:22:48 PM Answered Choice(s):B 
3 Choice 4:33:04 PM Final Choice(s): B 
1 Choice 4:04:22 PM Answered Choice(s): D 
2 Choice 4:33:04 PM Final Choice(s): D 
1 Choice 4:04:15 PM Answered Choice(s):C 
2 Choice 4:33:04 PM Final Choice(s): C 


1 
2 
3 
A] 
5 | 
=] 
7 
8 
2. 


3 
3 
4 
4 
5 
5 
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This structure of the snapshot log allowed to identify several micro-behaviors: viewing a 
question without selection, selection of an answer, change of an answer from correct to wrong, or 
change of a wrong answer to a correct one. These micro-behaviors were then used to defined 10 
overall behaviors that reflected potential sequencing across observed exam taker choices for the 
dataset used in this study. Examples of these behaviors include: correct answer from first choice 
to the final choice; view without a choice then correct answer to final choice; wrong answer as 
the first choice then correct answer to final choice; or wrong answer at the first choice, changed 
to correct answer and then wrong answer to final choice. The identified basic behaviors were 
mixed into behavioral patterns that included repetitive patterns. For the scope of this study, the 
behaviors were grouped in four outcomes, two beneficial and two damaging. 


The two beneficial behavioral patterns were: 
(bl-cc) correct - correct, correct answer from the beginning to the end of the exam 
(b2-wc) wrong - correct, wrong answer in the beginning changed into a correct answer by 
the end of the exam. 

The two damaging behavioral patterns were: 
(d1-cw) correct - wrong, correct answer in the beginning changed to a wrong answer 
(d2-ww) wrong - wrong, wrong answer from the beginning to the end. 


Qualitative Analysis Phase 


For the quantitative phase of the analysis, two exams in the same course that were 
administered during the COVID-19 pandemic were considered for this study. The first one was 
administered during the first stage of the pandemic when the face-to-face attendance was 
acceptable with distancing and class occupancy and masking requirements were strictly 
observed. The second exam was administered at the time when the pandemic requirements were 
tightened and remote instruction was in place. For each of the exams, seven high achieving 
students placed in both exams within top 20% of the grades as well as seven low achieving 
students placed within the bottom 20% of the scores were selected for this study. For each exam 
each student’s snapshots were coded at the assessment item level for the four observed 
behavioral patterns identified in the previous step and then the percentage of each behavior 
relative to the total number of assessment items was computed. 

In the next step, a synthesis Excel spreadsheet was created by recording for each student 
the percentages for each of the four observed behavioral patterns computed in the previous step, 
the maximum number of choices, the correct/wrong answer code, the grade level and the exam 
code (1-live, 2-remote). Figure 4 shows a snapshot of the synthesis Excel spreadsheet generated 
and used for this phase of the data analysis. 

Because of the small number of participants, a non-parametric test, Kruskal-Wallis, was 
used to compare the considered dependent behavioral variables and number of choices by exam 
type and respectively achievement/grade level. 
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Figure 4 
Synthesis Spreadsheet for Statistical Data Analysis 


ex_id grade level bi1_cc di_cw d2_ww max_choices 


StudID 

fs20 413 ex1 live high 1 
fs20 413 ex1 live high 2 
fs20 413 ex1 live high 3 
fs20 413 ex1 live high 4 
fs20 413 ex1 live high 5 
fs20 413 ex1 live high 6 
fs20 413 ex1 live high 7 
fs20_ 413 ex1_live low_1 


1 96.43 0.00 3.57 
1 
1 
7 
1 
1 
1 
1 
fs20 413 ex1 live low _2 1 
1 
1 
1 
1 
1 
2 
2 


96.43 0.00 3.57 
96.43 0.00 3.57 
96.43 0.00 3.57 
89.29 0.00 7.14 
100.00 0.00 0.00 
82.14 0.00 10.71 
46.43 3.57 | 35.71 
32.14 28.57 17.86 
64.29 0.00 35.71 
50.00 7.14 | 35.71 
46.43 7.14 | 39.29 
53.57 7.14 35.71 
60.71 3.97 | 39.44 
89.66 0.00 3.45 
68.97 0.00 0.00 


1 
2 
3 
4 
5 
6 
7 
8 | 
| 9 


fs20 413 ex1 live low 3 
fs20 413 ex1 live low 4 
fs20 413 ex1 live low_5 
fs20 413 ex1 live low 6 
fs20 413 ex1 live low_7 
fs20 413 ex2_rem_high 1 
fs20 413 ex2_rem_high 2 
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Findings 


When the observed beneficial behavioral patterns were analyzed by the student 
achievement levels, a Kruskal-Wallis test indicated that high achieving students ranked 
statistically significant higher in beneficial correct-correct behavioral pattern (21.5) than low 
achieving students (7.5), H (1) = 20.37, p <0.001. However, the two groups ranked about the 
same in beneficial wrong-correct behavioral pattern (p=.0.083). 

For the damaging behavioral patterns, the Kruskal-Wallis test indicated the same type of 
differences between low and high achieving students: 

- for damaging correct-wrong behavioral pattern the low achieving students ranked 

statistically significant higher (20) than high performing students (9), H (1) = 16.16, p 

<0.001; 


- for damaging wrong-wrong behavioral pattern, the low achieving students ranked 

statistically significant higher (21.5) than high performing students (7.5), H (1) = 20.59, p 

<0.001; 

When only the high achieving students were considered for the analysis, a Kruskal- 
Wallis test indicated a statistically significant higher ranking only for damaging wrong-wrong 
behavioral pattern during the live exam (10.1) when compared to the remote exam (8.9), H (1) = 
6.04, p<0.05. No statistically significant differences were found for the two positive behavioral 
patterns and second damaging behavioral pattern (correct-wrong). 
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However, when only the low achieving group was considered for the analysis, the 
Kruskal-Wallis test indicated: 
- a Statistically significant lower ranking for correct-correct beneficial behavioral 
pattern for the remote exam (4.4) when compared to the live exam (10.6), H (1) = 
7.65, p<0.01, and respectively 


- a Statistically significant higher ranking for wrong-wrong detrimental behavioral 
pattern for the remote exam (11) when compared to the live exam (4), H (1) = 
10.27, p<0.01. No statistically significant differences for the beneficial wrong- 
correct and detrimental correct-wrong behavioral patterns between the two types 
of exams. 

Finally, when the maximum choices by question were analyzed across the two types of 
exams (live and remote), the Kruskal-Wallis test indicated a statistically significant ranking for 
the remote exam when compared to the live exam for both the high achieving group (p<0.05) 
and the low achieving group (p<0.01). 


Discussions 


By expanding the use of ExamSoft snapshots from exam security purpose to the analysis 
of exam taker behaviors we were able to identify behavioral patterns and their variation across 
student achievement levels and across different examination contexts. The findings from this 
study indicated that, as expected, high achieving students made better answer choices and these 
choices were reflected in more beneficial outcomes while the low achieving students had 
behaviors that resulted in an increased level of damaging outcomes. 

When students were placed in an online exam format due to the COVID 19 pandemic- 
related constraints, students in low achieving group were more likely to exhibit behaviors that 
decreased the beneficial outcomes and increased the damaging ones. Finally, we found that 
during the online examination students in both achievement groups made more maximum 
choices by exam questions which can be explained through a combination of a potential increase 
in the level of anxiety and less distractions during the exam compared to the face-to-face context. 
However, additional research is needed to analyze the factors that impact, positively or 
negatively, assessment in a remote testing format. 
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Abstract 


A mixed-methods collective case study was conducted to assess how training and accompanying 
instruments impact teams’ implementation of Restorative Leadership skills in competitive and 
noncompetitive settings. Using a parallel mixed methods hybrid design where focus groups, mixed- 
methods surveys, observer checklists, one-on-one interviews, quizzes, and self-assessment checklists 
were used to address five research questions. Nine noncompetitive teams in two honors classes, a 
mock trial team, and an assistant mock trial team coach from a medium Southeastern university 
participated in this study. The nine noncompetitive teams were assigned to watch instructional 
videos about Restorative Leadership touchstones and were required to complete a 10-question quiz, 
followed by a voluntary Qualtrics mixed survey to get feedback on two restorative skills checklists. 
A leaderboard (dashboard) was implemented in the competitive mock trial team setting with two 
teams (Gold Team, Green Team). The purpose of this study was to determine if current Restorative 
Leadership instruction and accompanying tools can successfully aid both competitive teams and 
noncompetitive teams to implement Restorative Leadership skills in various activities. The results 
indicated positive support for the checklists to assess the application of Restorative Leadership skills 
for noncompetitive teams and mixed results for the impact of a leaderboard on individual and team 
performance for competitive teams. 


1. Introduction: 

What would it be like to create a model of leadership that could improve communication, 
accountability, and team relationships in a post-pandemic workplace? Where would we begin? Law 
schools across the country have primarily used the Case Method since 1890 to train teams for 
competitions (Chisholm, 1911). However, team cohesion, a key component in the success of team 
competition, has been undermined in traditional team training (Salas et al., 2015). Studies have 
shown that training focused on restorative leadership training improved participants’ 
communication, accountability, and relationship with other team members (Blankenship, Mwenja, 
Dolowitz & Wech, 2021; Pointer, 2019). Currently, the definition of restorative leadership is a work 
in progress that centers around four core tenets: working together to achieve invitation over 
coercion, radical inclusion, equitable communities, and working together to achieve objectives 
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(Blankenship, et al., 2021). The heart of restorative leadership is the push to help teams address 
challenges that naturally occur within a team over time. Yet the problem is how to offer restorative 
leadership training effectively. A recent study by Vegt, Visch, Vermeeren, and Ridder (2018) 
showed that gamification with team performance components was associated with increasing team 
cohesion. 


In this study, we introduced a new restorative leadership training model in several courses and 
examined its effectiveness on participants’ performance. This model was featured with gamified 
team strategy and restorative leadership training. The research questions were: 


1. How effective was the restorative leadership training in helping participants apply the 
skills in application exercises? 

2. How did the use of the gamified team strategy impact participants’ individual 

performance? 

How did the use of the gamified team strategy impact team performance? 

How did the use of gamified team strategy influence participants’ preparedness for class? 

5. How did the dual assessments impact participants’ performance? 


a 


2. Research Design 

This was a parallel mixed method collective case study. For the qualitative analysis, we examined 
four different cases simultaneously. We examined how teams implemented the Restorative 
Leadership model in team activities through self-assessments, observation of other teams, focus 
group interviews, and surveys. We used the quantitative analysis to examine the impact of the 
Restorative Leadership training model on participants’ performance, looking at the certificate 
completion, the self-assessment checklist, and individual quiz results. Qualitative results from the 
focus groups were used to supplement the results of quantitative analysis. 


2.1 Participants. Forty-seven undergraduate students in nine noncompetitive teams (23 in an honor's 
college honor's course with four teams, and 25 in a business honor’s course with five teams) and 30 
participants, of which 17 were active on two competitive teams according to the final leaderboard, at 
a midsized Southeastern University in the U.S. participated in this study. The students were 
primarily juniors and seniors. Participants in the focus groups were from these samples. The Gold 
Team focus group had three participants, the Green Team had four participants, and the honor’s 
college course focus group ended up being an individual interview. One individual interview was 
conducted with the Gold Team's assistant coach, a former Gold Team member. 


2.2 Intervention. Participants who were from the honors courses received the Restorative Leadership 
training. Students from the mock trial teams were divided into two teams: Gold and Green teams. 
The Gold Team were students who received the gamified team strategy and participated in mock 
trial as an extracurricular activity. The Green Team included students who only received the 
gamified team strategy in training, where participation on the mock trial team was part of their grade 
for their course. 


51 


Participants from the honors courses were to watch four Restorative Leadership instructional videos. 
Each video defined, described, and created a common language for each of the four tenents 
mentioned above. After watching the video, students engaged in roleplays during the class, where 
there are at least one to two observers on pre-established teams (in some cases, the whole team). 
After the roleplay, everyone was to complete either the self-assessment checklist on a Google Form 
or the Observer's Checklist on a Google Form. Then a debrief was conducted regarding the 
participants' experiences. This cycle occurred at least once more, where students who roleplayed 
now become observers and vice-versa. 


A leaderboard (dashboard) was created for the mock trial team for both Gold and Green team 
participants to see their standing as individuals and as teams. These dashboards were used to form 
the team that competed in collegiate competitions. The seven top performers, according to the 
dashboard, were selected. Points were earned based on individual performance in various roles 
during mock trial practices, learning materials, and appropriate application of the mock trial rules. 
Individuals and teams also earned points for demonstrating Restorative Leadership skills during 
mock trial training drills and practices. 


2.3 Data Collection. The American Mock Trial Association judging form (AMTA, 2019) was 
adapted into a Restorative Leadership assessment tool. This tool is developed into both a self- 
assessment checklist and an observer checklist. The self-assessment checklist included 14 behavior 
questions with a scale ranging from 0 (“Fails to Practice Skill”) to 8 (“Practices Skill and 
Encourages Others to”). During the semester, as part of the assigned course work, both honor’s 
courses completed the Restorative Leadership training videos, were required to provide a certificate 
of completion, and took a 10-question quiz about the materials. In addition to this, each participant 
was asked to complete a self-assessment checklist to assess their performance of the Restorative 
Leadership skills after engaging in a roleplay activity. The observer checklist included 14 questions 
with the same scales, and after roleplays where the participants were observers, they were asked to 
complete the observer checklist. Two focus groups were conducted with the mock trial’s Green and 
Gold Team members. The focus groups were designed to ask participants’ experiences and 
reflections on the use of gamified team strategy. 
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3. Results. 


Figure 1 
Design Model 


Parallel Mixed Methods Collective Case Study Design, 2021 
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apply the skills in application exercises? This question was to assess participants performance 
applying restorative leadership skills after receiving restorative leadership training. The 


noncompetitive teams, 47 of the 48 participants completed the individual quiz. The range was 5-10, 


the average was 7.85, and the standard deviation was 1.54. Over 76% scored above a seven, and 


about 30% scored nine or higher on this quiz. These results indicate that students could recall 50% or 


more of the key concepts from the Restorative Leadership videos. Of the 48 students required to 
complete the training, 47 students participated in all or most of the activities, 23 were required to 


submit a certificate of completion as part of the module requirements (honor’s college course). This 


certificate was only obtained after they completed quizzes built into the training. 19 of these 23 
submitted a certificate. In the business honor’s course, 24 of the 25 who participated earned the 
certificate. To receive a certificate, participants had to score higher than 80%. 


Research questions 2-4 were specific to the competitive teams because they addressed the influence 


of the gamified team strategy, a leaderboard/dashboard, on participants’ performance and 
preparedness both individually and as a team. Quantitatively this was to be assessed by the mock 
trial restorative checklists and an anonymous mixed-methods Qualtrics survey. These Quantitative 
results were to be augmented with two focus groups, one with each team. However, only ten mock 
trial members completed the survey, four Gold Team and six Green Team members. Because this 
was anonymous, it is also possible that some of these participants also participated in the focus 


groups. 


53 


Research Questions 2. How did the use of the gamified team strategy impact participants’ 
individual performance? This question was specific to the competitive teams to see how the 
gamified strategy impacted their individual performances. Though there were only ten members, the 
results showed that 90% paid some attention to the dashboard (yes, somewhat, no), and only one 
member did not Mean = 4.70, SD = .64. On a 5-point scale (strongly agree to strongly disagree), 
70% of the participants felt that the dashboard somewhat too strongly influenced their individual 
performance, Mean = 2.30, SD = .78. When exploring how the dashboard influenced their individual 
preparedness, the participants reported the dashboard somewhat to strongly influenced their 
preparedness, Mean = 2.20, SD = .76. 


Research Question 3. How did the use of the gamified team strategy impact team performance? 
This question was specific to the competitive teams to see how the gamified strategy impacted their 
team’s performances. When asked to assess the influence of the leaderboard on team performance, 
there was more of a divide. 40% somewhat agreed, 20% remained neutral, and 40% somewhat 
disagreed, Mean = 3.00, SD = .89. The results for how the leaderboard influenced the team's 
preparedness were varied, 40% strongly or somewhat agreed, where 40% somewhat or strongly 
disagreed again 20% remained neutral, Mean = 3.00, SD = 1.18. 


Research Question 4. How did the use of gamified team strategy influence participants' 
preparedness for class? This question was specific to the competitive teams to see how the gamified 
strategy impacted their individual preparedness for class. When exploring how the dashboard 
influenced their individual preparedness, the participants reported the dashboard somewhat to 
strongly influenced their preparedness, Mean = 2.20, SD = .76. 


Research Question 5. How did the dual assessments impact participants' performance? 

This question was to assess participants perceptions and use of the restorative leadership checklists. 
At least 38 participants responded to portions of both checklists. The responses were not identical for 
both the self-assessment checklist and the observer checklist. Of the 38 participants who completed 
the self-assessment checklist, 37, or over 97%, stated they understood what was being asked of 
them, 20, and understood their response options, 17. For the observer checklist, 21 of the 
participants, just over 55%, said they understood what they were being asked, 15, over 39% said 
they understood their response options, one participant, 2.6%, said they did not know what they were 
being asked. One participant marked other, 2.6%. When examining the checklists, there is a 
difference between the self-assessments and the observation checklists. See Figure 2 for a 
comparison between two checklists. 


Qualitative Results for Research Question 5. The decision not to run any statistical tests was made 
because of the qualitative feedback from the mixed methods Qualtrics survey. A common theme that 
emerged from participants regarding the self-assessment was changing the formatting, reordering the 
questions, putting an N/A option, or including open-ended questions. However, just as many 
responses stated that no changes were needed, it was clear and concise. When asked what reason 
participants would recommend, or not, recommend the self-assessment for assessing leadership 
skills, the predominant theme was that participants would recommend this assessment because it 
helped participants reflect on their actions, “I would recommend the self-assessment because it 
allows one to reflect and become aware of one’s past actions in order to make improvements.” 
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Another theme that emerged was how this tool provided a way for participants to be a better leader, 
as noted by the following two quotes. “I think the self-assessment overall is a good tool because not 
only does it remind you of the leadership skills, but it also provides a checklist of actions that you 
can evaluate yourself on and maybe encourage you to do better in the future." And “I would 
recommend simply because I have learned so much from it of how to be a better leader, get out of 
my comfort zone when it came to talking about personal problems from the past, and just be more 
supportive others throughout this process.” 


The Observer Checklist had contradictory themes and seemed to cause participants confusion. Based 
on responses, it appeared that instructions were not clear on how to use the Observer Checklist. 
Participants were not sure if they were to focus on each individual in the role play or the overall 
efforts or objectives of the exercise. Some participants felt that the identified skills did not go with 
the activity they were observing, “I felt like some of the questions did not apply to the situation, so I 
was confused on how to respond. Also, clearer instructions would have helped to know how to 
answer the question." Of the 21 responses to what participants would change, 13 stated "NA," which 
indicated that they had read the question and had no comments about changes. One participant 
recommended that we have a comments section about what observers enjoyed about the activity they 
were observing. 


When the participants were asked if they would or would not recommend the Observers’ Checklist, 
like the self-assessment, the common theme was yes. Participants felt that this assessment enabled 
them to “properly” assess the players in the role play. Others found it to be helpful in a social 
learning context, “I would recommend this survey because by reflecting on what others did well and 
what they could have improved upon, you can improve your own approach. You may see some 
similarities between yourself and others and decide that didn't work out for them, so I'm not going to 
do it either." Of the 21 responses, two participants stated "NA," again indicating they had read the 
question, one person said, “I did not use it but they people on my team that did helped us see what 
we did not,” and one person said they were not sure if they would recommend it because they feel an 
open-ended assessment would be more “valuable way of reflecting.” 
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Figure 2 
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Note. The cells are split for ease of comparisons between self-assessments on the Restorative Leadership skills 
and observers' assessments of applied Restorative Leadership skills during roleplays. The number reflects the 
number of participants that marked those specific ratings. 


Qualitative Results, Focus Groups. Though a focus group for the noncompetitive team did not 
occur, the individual interview supported the theme of confusion regarding instructions. The 
participant did make it clear that she enjoyed learning about Restorative Leadership. However, when 
asked if her team was using restorative skills, she stated that the team was not using them in their 
interactions. Then later, she stated that the restorative leadership training was helping the team 
understand the different ways they communicated. 


A major issue was discovered too late in the study. The program director and coach of the Green 
Team did not administer the Mock Trial Restorative Checklists as the teams' dual assessments. He 
created his own assessment. Based on the low number of participants and the feedback from the 
focus groups, these assessments were found to be confusing as time went on during the semester, 
and members from each focus group stated that they were confusing and did not help performance or 
preparation. The following two quotes exemplify what was reported in both the survey and the focus 
groups: “I don't think that I saw any changes after the implementation of the assessments. I think 
most of the team side as a chore and they were confused 90% of the time that they were filling it out, 
and the only feedback I could give reach out to the Professor [blank]or try your best because,” and “I 
don't know if they necessarily help because, like a lot of us just ended up working with people that 
we already kind of knew how they worked in a sense, and so like. It was definitely a..... I guess a 
little weird like kind of going in and like sometimes I felt like seeing the assessment question did 
like apply, but also didn't apply and so.” 


These quantitative findings are supported by what was discussed in both focus groups. The results 
were mixed. However, there was a definite difference between the Gold Team and Green Team 
themes. The Gold Team's introduction to the leaderboard caused the team confusion, but they also 
felt it was mismanaged and not transparent. It caused them to focus on their individual performance 
at the cost of team cohesion. Two participants stated, "I was worried that they would then come in 
and take a spot, and then they would take away the team chemistry and it wouldn't work as well, and 
not - not even mentioned the fact they would also have to learn the material and a much more quick 
rate and may not be able to catch up as fast.” And, “You know, but instead we were kind of focused 
on kind of our own individual things that we had to do so that we could just get on the team to begin 
with, with you know what felt like kind of an arbitrary leaderboard system. So, it definitely kind of 
forced us to working on our own when we should have been working together.” The Gold Team’s 
coach’s comments also indicated this dual result, “the gamification piece was helpful in the in the 
way that it incentivizes them to work harder. And it pushed them it basically gave them a reason to 
kind of go further than what they go beyond what they use what.” 


When asked what surprised the Gold Team participants the most, one of the focus group participants 
stated, ““What surprised me most was how much better prepared, we were this year for the 
competition, we were last year um it was it seemed like we were much more. We had a much better 
grasp of the material; we knew what we were doing a lot more.” Overall, the themes for both the 
assistant coach and Gold Team members reflect how the leaderboard did impact their individual 
performance and helped their preparedness. However, initially, it was at the cost of the team 
cohesion, or what this focus group called team "dynamics." 


57 


On the other hand, the Green Team felt that the leaderboard helped provide them feedback on their 
performance and what they needed to earn points and be better prepared. They also mentioned how 
Canvas impacted their preparation. This latter theme was labeled under “other” in the Tools to 
Impact Performance theme. This is because a key difference between the Gold and Green teams’ is 
that the Gold Team is voluntary, and the Green Team comes from a mock trial course that students 
are getting course credit for. 


The results from both the quantitative portion of this study and the qualitative indicate that the 
Restorative Leadership training helped help participants recall and apply restorative skills during 
their quizzes for the certificate and the 10-question quiz given in the class as part of their Restorative 
Leadership Module. In addition to the quizzes, the mixed methods survey regarding the checklists 
overwhelmingly said these checklists helped evaluate and apply their skills, despite some confusion 


regarding labeling of skills and instructions for the observer checklists. 
The quantitative and the qualitative data indicate that the leaderboard did influence both 
performance and preparedness, individually more than the team. It appears that the Green Team 
benefited positively from having the leaderboard, both as feedback on their performance and as 
guidance on what to do. For the Gold Team, they felt that the leaderboard harmed their team 
dynamics initially. As time went on, they felt that it could be a good tool, with more transparency 
and better management. See Figure 3 for a summary of the outcomes for the research questions. 


Figure 3 
Research Question-by-Outcome Joint Display 
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checklist responses main concepts and how to use the 
results. recommended no from the videos. | checklists during 
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performance 
and 
preparedness 
than on team 
performance 
and 
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also support the 
mixed results. The 
main themes here 
were that 
participants felt 
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however many 
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and they felt it did 
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leaderboard could 
be effective if it 
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well and how 
points were 
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However, they 
felt that other 
things could 
determine a 
successful team, 
such as observing 
the dynamics of 
individuals and 
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coach. For the 
Green Team, 
overall, they 
found the 
leaderboard 
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points motivation. 
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management and 
that the leaderboard 
alone does not help 
with team 
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performance but 
appears to impact 
individual 
preparedness and 
performance more. 
There appear to be 
two emergent 
themes that should 
be examined further, 
the role of 
motivation and 
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performance and 
preparedness. 
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4. Discussion. Several issues emerged when recruiting participants for the focus groups, for both the 
competitive teams and noncompetitive teams. It was clear that the primary researcher needed to be 
the one who sent all communications to the participants, such as making the requests to participate in 
the focus groups, completing the Restorative Leadership Checklist assessments, having the Gold 
Team review the Restorative Leadership instructional videos, and ensuring that the competitive team 
participants received invites and follow-ups to complete the Qualtrics survey. By not having the 
primary researcher in charge of the above processes, valuable input was lost from having neutral 
outside observers use these checklists to evaluate the competitive teams' mock trial skirmishes or 
competition. This insight would have given the study more data about how to revise the Restorative 
Leadership Checklists for clarity as well as helping determine how the current revised checklists’ 
reliability and dependability were viewed by an individual who is unbiased and knowledgeable 
about mock trial processes. 
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Three interesting themes emerged during the focus groups: the role of motivation, experience vs. 
nonexperience, and how the Gold Team felt that the dashboard did little to improve the Green 
Team's performance as the semester wore on. The program director has begun redesigning the team 
so that motivation and experience levels might be offset because a mock trial class will no longer be 
offered. He is determining what the prerequisites will be for joining the mock trial team. He is 
determined to keep the leaderboard, and hopefully, the results from this study will help him create 
more clear guidelines and management with the leaderboard. 


The significance of this study is that once these instruments prove to be valid and reliable in settings 
beyond a mock trial course, they can be used to help train a workforce in Restorative Leadership and 
ensure that employees are transferring the skills to their workplace. The results of this study are 
promising about the possibility of interdisciplinary application of the Restorative Leadership 
instructional videos, Restorative Leadership application exercises, the Restorative Leadership 
checklists as assessments of applied skills, and does warrant more in-depth studying. 


4. Limitations and Future Directions 


There were many limitations to this study. One major limitation was that convenience sampling was 
used, which meant that one of the noncompetitive teams was co-taught by the primary researcher, 
the program director from the mock trial team, and another professor. This could have produced 
response bias, researcher bias, and some threats to internal validity, such as timing or maturation. 
Not only is this study subject to these biases and threats, but the sampling process, which was a 
combination of convenience and purposive, also makes generalizing our results difficult. Because 
there were unplanned changes to the design that impacted research questions one and five, not to 
mention the low number of participants for research questions two, three, and four. 


These limitations, however, guide numerous directions for future research. After revising the 
observer checklist, it will be good to test the instructions and ensure that the identified skills are 
observable. Next, the videos should be evaluated using Kirkpatrick’s four levels of evaluation 
(Kirkpatrick, 2006). The researcher will want to evaluate the reaction and learning levels. A mixed- 
methods survey could help assess if and how the videos helped apply restorative leadership skills. 
Finally, another area to explore more is how participants are motivated and how they view either the 
leaderboard and/or the Restorative Leadership training.4. Significance of Research 


The significance of this study is that once these instruments prove to be valid and reliable in settings 
beyond a mock trial course, they can be used to help train a workforce in Restorative Leadership and 
ensure that employees are transferring the skills to their workplace. The results of this study are 
promising about the possibility of interdisciplinary application of the Restorative Leadership 
instructional videos, Restorative Leadership application exercises, the Restorative Leadership 
checklists as assessments of applied skills, and does warrant more in-depth studying. 
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Introduction 


Institutions of Higher Education (IHEs) are focusing on online programs as a way to 
innovate (Magda & Buban, 2018), but faculty buy-in is a major barrier (Walker et al., 2018). 
Due to the COVID-19 pandemic in Spring 2020, many higher education faculty were forced to 
teach online for the first ttme (Means & Neisler, 2020). Rogers’ (2003) Diffusion of Innovations 
(DOI) theory suggests that the adoption of an innovation, such as online teaching, takes place 
over time, and the last group to adopt are known as laggards. They adopt an innovation often 
when there is no other choice. In this study, the term “pandemic laggards” will be used to 
describe faculty who started online teaching due to the pandemic. 

Individuals undergo a process in deciding to adopt (or not adopt) an innovation, which is 
influenced by multiple factors (Rogers, 2003). However, two factors are particularly relevant to 
this study. The first, “re-invention” is changing or modifying an innovation in the process of 
implementing it (Rogers, 2003). Higher degrees of reinvention tend to speed adoption and lead to 
longer use (Rogers, 2003). The second factor is organizational culture (OC) or the “system of 
beliefs, values, and behavioral norms that come to be taken for granted” (Schein & Schein, 2016, 
p. 6). With regard to technology use, aspects of OC have been shown to be influential among 
faculty (Boichuk & Fast, 2017; Reid, 2017) and in implementing online (Zhu, 2015; Zhu & 
Engels, 2014) and blended learning (Porter & Graham, 2016). 

The pandemic provided an opportunity to explore the relationship between OC and online 
teaching adoption among a unique group of faculty adopters. The purpose of this sequential 
explanatory mixed methods study was to explore how pandemic laggards’ willingness to adopt 
new online teaching strategies were related to their views of their departmental culture. 


Conceptual Framework 


The conceptual framework for this study combined Rogers’ (2003) DOI to categorize 
faculty adopters, and Cameron and Quinn’s (2006) Competing Values Framework (CVF) to 
describe OC. The CVF and its diagnostic component, the Organizational Culture Assessment 
Instrument (OCAJ), was designed to capture participants’ views of their current and desired 
culture into one of four culture orientations (Figure 1): clan, adhocracy, market and hierarchy 
(Cameron & Quinn, 2006). The OCAI is organized in six dimensions: Dominant Characteristics, 
Organizational Leadership, Management of Employees, Organizational Glue, Strategic 
Emphases and Criteria for Success. 
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Figure 1 
Competing Values Framework 
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Note. Reprinted from Diagnosing and Changing Organizational Culture (p. 35), by K. Cameron 
and R. Quinn, 2006. Jossey-Bass. 2006 by John Wiley & Sons, Inc. 


Clan organizations engender loyalty through shared values, consensus decision making, 
and practices that empower employees (Cameron & Quinn, 2006). Hierarchy organizations 
foster stability and efficiency and value clear lines of leadership, standardization of processes, 
and employee accountability (Cameron & Quinn, 2006). Organizations with a Market culture 
focus on the external environment, with the constant goal of gaining a competitive advantage 
(Cameron & Quinn, 2006). Adhocracy organizations have a decentralized structure that values 
individuality, experimentation and future-oriented thinking (Cameron & Quinn, 2006). 


Context & Design 


This study took place in Spring 2021 at a flagship campus of a large public research 
university, which typically offers approximately 500 courses via distance education per semester. 
In Spring 2020 prior to the pandemic, about 15% of courses were taught via distance. In Fall 
2020, 81.6% of courses were taught online, 12.9% were hybrid (combination of in-person and 
online) and just 5.5% were face-to-face (Bruno, 2020). 

Email surveys were sent in February 2021 to faculty who taught online that semester 
(n=1281), which asked about participants’ online teaching background and departmental culture 
based on the OCAI. Of the 184 usable responses, 61% of faculty (n=113) reported being new to 
teaching online due to the pandemic. 

Additional data were collected via seven follow-up interviews with selected survey 
respondents who identified themselves as new to teaching online due to the pandemic, and who 
characterized their department with varying OC orientations. Interviews were analyzed using 
thematic coding (Ary et al., 2019) to obtain codes, categories, and final themes. 
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Results 


Quantitative 

The quantitative portion of this study sought to answer, what is the relationship between 
pandemic laggards’ current and desired OC and their plan to teach online once the pandemic is 
over? 


Current & Desired Cultures 

Of the 113 self-identified pandemic laggards, 45% (n=51) described their current departmental 
culture as a Clan type, 35% (n=40) as Hierarchy, 9% (n=10) as Market type and 5% (n=6) as an 
Adhocracy. In addition, 5% (n=6) described their culture as a mixture of multiple types with four 
of those describing their OC as a mixture of Clan and Hierarchy. 

As for participants’ desired culture, 66% (n=74) desired a Clan culture, 14% (n=16) an 
Adhocracy, 10% (n=11) a hierarchy, and 3% (n=3) a market culture (Figure 2). Seven percent 
(n=9) desired a mixed culture, with eight preferring a mixture that included the clan culture and 
six including the hierarchy culture. 


Figure 2 
Pandemic Laggards’ Current and Desired Cultures 


Current Culture 
Desired Culture 
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a Mixed 
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Post-pandemic teaching plans 

Of the 104 participants who responded to this question, sixty-two percent were willing to teach 
online after the pandemic was over (n=64), while 19% (n=20) were undecided or neutral, and 
19% (n=20) were unwilling to teach online (Figure 3). 
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Figure 3 
Plans to teach online post-pandemic 
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No significant relationships were found between participants’ current and desired cultures and 
their plans to teach online post-pandemic. A chi-square test found no significant association 
between current culture and pandemic online teaching plans X? (8, N = 104) = 9.91, p= .271 
ando no significant association between desired culture and pandemic online teaching plans X” 
(8, N= 104) = 8.80, p = .360. 


Qualitative 

The qualitative portion of this study explored how pandemic laggards described the influence 
their OC had on their willingness to adopt new online teaching strategies and their plans to teach 
online after the pandemic. Pseudonyms are used for interviewees. 


Willingness to adopt new online teaching strategies 

There were generally two levels of faculty willingness to adopt new online teaching strategies, 1) 
transformational and 2) minor modifications. Four interviewees made transformational changes, 
taking advantage of a number of online tools and/or strategies. Examples of these changes 
include opting for an asynchronous delivery mode, and creating an online set of modules to 
accompany a course. Three interviewees only made minor modifications, delivering their courses 
largely lecture-based using web conferencing tools such as Zoom. These faculty made only 
minor modifications to how they teach in person. The OCAI’s six dimensions were used to 
organize the emergent themes. 

Dominant Characteristics. Dominant characteristics describe the overall characteristics 
of an organization's culture. The interviewees described their department cultures in different 
ways, see Table 1. One important theme that emerged was individualistic, as the faculty 
interviewed were primarily from departments in which collaboration around teaching and 
research was not the norm. Participants were split when describing their departments with three 
describing their department’s culture in mostly positive terms, and four describing their culture 
in critical terms or as less than ideal. For the three who were generally positive, they used terms 
like collegial, supportive and personal. Two out of three were from clan cultures, and all three 
chose clan as their desired culture. For the four who were critical, they used terms like 
inefficient, dysfunctional or transitional, and three out of four desired cultures that were different 
from their current culture. 
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Table 1 
Interviewees’ cultures 


Interviewee Positive or Current Desired Descriptors of Culture 
Critical Culture Culture 
Beau Positive Hierarchy Clan Individual with shared decision making 
Nathan Positive Clan Clan Performance focus, mixed personal connection 
Cory Positive Clan Clan Collegial and supportive 
Patricia Critical Market Clan Supportive leadership, but stressful due to many changes 
Fiona Critical Hierarchy Adhocracy Dysfunctional, conflicting personalities and values 
Ethan Critical Clan Adhocracy _ Inefficient, Culture of Poverty 
Martin Critical Hierarchy Hierarchy Highly individual, divided by personal agendas 


Leadership Style. Leadership style is the approach to leadership within the department. 
In this area, two themes emerged. The first theme was that of responsive leadership, particularly 
in regard to instructional issues. Examples of positive or responsive leadership included 
providing teaching and learning assistants to help faculty with the increased workload, 
advocating to unit or campus-level administration about instructional needs, emotional support 
when faculty came to them with issues, and providing online teaching resources such as training 
or materials. Examples of negative leadership included being too closely aligned with 
administration-level politics or being too focused on short-term financial gain versus capacity 
building, both seen as detrimental to an increase in online teaching with one interviewee stating, 
“Well, I mean given that there's really... a culture of not having any extra resources or ability or 
investment in pedagogy, that doesn't really make me want to try out a lot of stuff.” 

Another theme that emerged was faculty autonomy. Most interviewees described their 
department’s leadership as fairly hands off in terms of how faculty taught their courses. 
According to a number of interviewees, a high degree of faculty autonomy contributed to their 
willingness to try new online teaching strategies, as they felt free to make changes according to 
their preferences or student needs. One interviewee mentioned that prior to the pandemic her 
course enrollments were closely scrutinized and her courses often closed due to low student 
numbers. During the pandemic, she noticed less oversight, which contributed to her feeling more 
freedom to implement new online teaching strategies. Some interviewees mentioned that because 
of a high degree of autonomy, they did not think culture impacted their willingness to adopt new 
online teaching strategies. 

Management of Faculty and Staff. This dimension encompasses how employees are 
treated and looks at the working environment. In this dimension, two themes emerged. The first 
is efficient administrative support. Several interviewees expressed being stretched thin during the 
pandemic, impacted by both a pandemic hiring freeze and increased demands of pivoting to 
online learning. One faculty member mentioned that it took weeks to handle simple purchasing 
with grant funds, inefficiencies that took up time that could have been spent on teaching 
practices. 
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The second theme was targeted instructional support. Several interviewees mentioned 
utilizing campus or unit-level support in their transition to online teaching. However, existing 
services did not meet all interviewees’ needs. One interviewee desired personal support saying, 
“T didn't have the time or the energy to explore all the technical possibilities and exchange five 
letters with ITS [campus IT services]. If there was somebody here... it would help.” In addition, 
there were particular issues faced by interviewees who taught large classes, such as giving 
exams, grading and preventing cheating. Two interviewees mentioned the importance of 
discipline-specific online teaching support. Beyond technical or instructional support, two 
interviewees mentioned other practical needs. One lacked a proper space at home to conduct 
Zoom sessions, and another interviewee was not able to be reimbursed for teaching software due 
to campus rules on reimbursements. 

Organizational Glue. Organizational glue is what bonds an organization together. In this 
area, three themes emerged. The first was students, as stated by one interviewee, “Organizational 
glue that binds our department together, really that's our students because everyone cares about 
our students quite a lot.” All interviewees talked about how students’ feedback and course 
outcomes influenced their practices. 

The second theme was peer support. Two interviewees spoke about influential peers in 
their departments who served as a hub for online teaching and resources or helped other faculty. 
Another interviewee shared that she herself provided peer support, giving demos on using Zoom 
and sharing with her colleagues a list of online teaching resources she had collected. She desired 
for peer support to become an established norm, stating, “I think, ideally, it would be nice...if we 
need something from each other, I feel comfortable picking up the phone or shooting you an 
email, you should feel the same way.” 

The third theme was collaboration. As mentioned, the interviewees primarily came from 
individualistic departments, but faculty described ways collaboration could influence their online 
teaching practices. Several interviewees pointed out that they felt more disconnected from fellow 
faculty due to social distancing. Co-teaching was a potential form of collaboration mentioned 
with excitement by one interviewee who stated, “That could work if we have somebody that's 
taught the course once and then they're bringing on additional people to help bring them up to 
speed using tools, but then also to bring in new ideas.” Collaboration also had the potential to 
negatively impact efficiency according to one interviewee who stated, “It makes no sense to 
coordinate with anybody else, that's only extra work in terms of the teaching.” 

Strategic Emphasis. Strategic emphases are areas of emphasis that direct the 
department’s strategy. In this area, one major theme emerged, the alignment to shared values and 
goals. One important shared value among all of the interviewees was student success. Other 
shared values and goals discussed were that of staying accredited, serving students in a culturally 
appropriate manner, reaching underserved communities and raising funds. Some interviewees 
discussed that an important shared value was to do what is best for the group or department over 
the individual. One interviewee mentioned that during the pandemic, he had been willing to take 
on additional class sections to benefit the department. 

Criteria of Success. Criteria of success is how success is defined and what gets rewarded 
and celebrated in the department. Two themes emerged in this area. The first was research, and 
the fact that it often is considered more important than teaching in faculty evaluation processes. 
Research was also discussed in terms of the amount of time it takes to apply for grants, publish 
in journals and supervise graduate students. 
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The next theme was departmental survival. Some interviewees described their 
departments as struggling due to financial strains or external pressures. In light of these issues, 
anything that promoted departmental survival was to be celebrated. One interviewee said, 
“Definitely student retention gets rewarded, and really anything positive that can put a positive 
light on our department.” 


Post-pandemic plans to teach online 
Regarding plans to teach online post-pandemic, there were two main themes. 

Face-to-face is primary. First was that face-to-face teaching should remain the primary 
delivery mode, but could include online elements. All interviewees wanted to move back to face- 
to-face teaching, however, six out of seven wanted to retain some online elements. Interviewees 
felt their asynchronous teaching materials such as online learning modules and discussion boards 
offered greater efficiency and could be reused in their courses post-pandemic. Three 
interviewees mentioned wanting to teach in the future using hybrid delivery methods, with a 
mixture of online and face-to-face sessions, or ‘““Here-or-there” sessions in which students have 
the option to attend face-to-face or online. Two interviewees were strongly considering keeping 
some of their courses online, despite wanting to move back to face-to-face delivery. Having gone 
through the experience of teaching online due to a crisis, all faculty expressed willingness to 
teach online again if absolutely necessary. 

While there was openness to hybrid online delivery among the faculty, most of the 
departments represented in this study did not have plans in place to offer new online courses or 
programs long term. One interviewee shared that his department was open to online courses and 
programs as a way to support more students, but the department needed help with developing 
guidelines, expectations and procedures. One interviewee shared that once his department 
decided to move to all online classes, faculty worked independently thereafter and never 
reconvened to discuss strategies or outcomes. 

Lessons learned. The second theme was lessons learned. Some of the interviewees were 
surprised by the effectiveness of online teaching. One interviewee remarked that in his 
asynchronous class discussions, he saw a greater depth of discussion than in any of his classes 
prior. Another interviewee experienced remarkable improvement in her students using the 
flipped classroom model, “My students are speaking, they're speaking the target language, like 
it's phenomenal...I have not seen this kind of progress.” 

Many of these “pandemic laggards” were also feeling more confident in their online 
teaching. One interviewee described it this way, “...having spent the time learning more about 
this in a crisis situation, you know, now I know a lot more...I could apply that in a much more 
intentional way.” Interviewees described multiple strategies they had used to address online 
teaching obstacles. For example, one interviewee shared that to address challenges of time 
management and increased enrollments, he implemented a series of short papers throughout the 
semester as opposed to one large term paper he previously assigned. Others focused on student 
differentiation strategies that could be employed online to address differing student needs, for 
example, employing Universal Design for Learning principles. Faculty also learned overall 
lessons about teaching that can apply in all situations. One interviewee described it this way, 
“I've again, taught for 15 years, and I've learned more about teaching in this past year, then 
maybe most of the years prior.” 
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Discussion 


This study explored pandemic laggards’ adoption of online teaching at one institution 
through the lens of OC. The majority of respondents described their departments as a clan culture 
and many also preferred this culture. This is consistent with findings from other higher education 
institutions (Obendhain & Johnson, 2004) which also tended toward clan culture. Perception of 
culture (culture orientations) did not have a statistically significant relationship to willingness to 
teach online post-pandemic. Due to social distancing related to COVID-19, faculty expressed 
feeling more disconnected from their peers and not aware of what each other were doing. 
Because OC is a shared phenomenon (Groysberg et al., 2018), it may have had less influence due 
to the effects of the pandemic. 

Among the seven interviewees’ culture orientations, there did not appear to be a clear 
relationship between current and desired cultures and levels of re-invention or modification of 
courses. Two interviewees believed their culture did not influence their online teaching practices 
due to the autonomy faculty have in their departments. This may suggest that in terms of online 
teaching adoption in higher education, that individual differences are more influential than 
departmental culture. This would align with the finding that individual innovativeness, or the 
desire to try new things, has been shown to be related to acceptance of instructional technology 
(Akgiin, 2017). Some transformational faculty, those who made major modifications to their 
courses, were indeed more willing to try new things in terms of their teaching. Another 
individual difference possibly influencing the results is help seeking, particularly the willingness 
to engage in professional development activities such as training or peer support. Daumiller et al. 
(2021) found that faculty who were motivated by gaining personal competence, had a more 
positive attitude to the shift to pandemic-related online teaching. 

The qualitative data, however, showed that aspects of culture also influenced willingness 
to adopt new online teaching strategies and willingness to teach online post-pandemic. While 
faculty were typically autonomous in how and what they taught, there were cultural factors of 
departments that influenced their teaching practices. Faculty were extremely busy during the 
pandemic (Giovannella & Passarelli, 2020), and factors that promoted efficiencies such as 
responsive leadership, and administrative, technical and peer support had an impact. Faculty also 
cared deeply about their students’ success and were willing to integrate new strategies that 
supported student success. Because a primary evaluation criteria for many of the faculty in this 
study was research, the time and resources they were able to devote to teaching was limited. 
These findings highlight a critical discrepancy between faculty evaluation criteria and 
institutional goals to grow online offerings and support instructional approaches that increase 
student success. Reid (2014) also identified rewards and incentives as a barrier to faculty 
adoption of instructional technologies. Addressing this discrepancy may be key to long term 
adoption of online teaching. Faculty are often willing to start new initiatives, but IHEs cannot 
sustain the effort without aligning the desired activity to reward systems (Farmer, 1999). 

Hodges et. al, (2020) assert that in Spring 2020 when the pandemic began, faculty were 
engaged in “emergency remote teaching,” characterized by less time for planning and quality 
assurance, and teaching with a one-size-fits-all approach. Means and Neisler (2020) projected by 
Fall 2020, with the proper supports in place, faculty would move toward optimal online teaching, 
which is characterized by effective planning, use of a developed instructional design process, 
integration of community and engagement, more time and support for quality assurance, and 
ensuring equity and personalization. This study found that some faculty new to teaching online 
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have begun to make the transition to optimal online teaching. While many felt stressed during the 
onset of the pandemic, several have integrated community, engagement and personalization and 
look forward to more time for planning. Continuous innovation, or the process of re-invention 
(Rogers, 2003), did in fact take place for nearly all faculty in this study. Faculty made various 
levels of modifications in their online courses in response to student needs, and many learned 
valuable lessons through the process. 

While nearly all interviewees wanted to keep teaching online or use elements of online 
delivery, they were excited to go back to face-to-face teaching. Faculty in a 2019 study 
expressed similar preferences, with only 9% preferring fully online teaching, and 51% favoring 
hybrid approaches (Galanek & Gierdowski, 2019). Now that many more faculty have been 
forced to teach online, these numbers are likely higher. Trialability, or the ability to try a new 
innovation, is a factor that impacts the rate of adoption (Rogers, 2003). 

It is important to note that all the interviewees were willing to teach online if needed, 
although it was not their preferred mode. The pandemic appears to have taught faculty that 
despite previous experience, they are capable of teaching online, and could do it again. Because 
of the experience of teaching online in response to a crisis, faculty appear to feel a sense of duty 
toward teaching online. This could be a fruitful area of future research. IHEs may experience 
more faculty buy-in if online teaching supported growth or survival, such as raising funding or 
increasing student access. Another potential area of future research could focus on the impact of 
departmental culture on the offering of online programs, versus faculty’s individual adoption of 
online teaching. Interviewees mentioned that within their departments online programs had been 
discussed but not implemented due in part to cultural influences such as conflicting values 
between administrators and faculty. Students want to keep taking fully online and hybrid courses 
post-pandemic (McKenzie, 2021). Now that faculty are open to teaching online or hybrid, it 
behooves departments to design course offerings that align with student preferences. 


Limitations 


This study was conducted during the COVID-19 pandemic, therefore results may not be 
generalizable to other periods. While OC has been shown to have a positive relationship to 
technology-enhanced and online learning, factors outside of OC frameworks could influence the 
results (Zhu & Engels, 2014). Also limiting generalizability is that adoption research is often 
based on self-reporting and reliant on participants’ memories (Rogers, 2003). In addition, 
because this research took place at one institution in a unique geographic and cultural context, 
results may not be transferable. The researcher is an employee at the institution under study and 
thus may also be prone to bias. 

Conclusion 


Higher education innovation experts assert that the lessons learned during the pandemic 
may lead long term to increased adoption, lowered costs and improvements in online teaching 
quality (Arum & Stevens, 2020). To take advantage of this transition, Fox et al., (2020) 
recommends IHEs, “use the momentum of this watershed moment to elevate your approach to 
online and hybrid instruction” (p. 4). This study has shown that faculty worked hard to transform 
their online instruction and look forward to making continuous improvements. It also revealed 
how aspects of OC, a key driver of innovation, can support this process now and into the future. 


71 


References 


Akgiin, F. (2017). Investigation of Instructional Technology Acceptance and Individual Innovativeness of 
Academicians. Turkish Online Journal of Qualitative Inquiry, 8(3), 291-322. 
https://doi.org/10.17569/tojqi.292135 

Arum, R., & Stevens, M. L. (2020, March 18). Opinion | What is a college education in the time of 
coronavirus? The New York Times. https://www.nytimes.com/2020/03/18/opinion/college- 
education-coronavirus.html 

Ary, D., Jacobs, L. C., Sorensen, C. K., & Walker, D. A. (2019). Introduction to research in education 
(Tenth edition). Cengage. 

Boichuk, P. M., & Fast, O. L. (2017). Organizational culture and technology-infused management in 
higher education: Theoretical and empirical aspects. Information Technologies and Learning 
Tools, 61(5), 219-232. https://journal.1itta.gov.ua/index. php/itlt/article/view/1764/1247 

Bruno, M. (2020). UH Manoa COVID-19 plan for fall. University of Hawaii News. 
https://www.hawaii.edu/news/2020/07/22/uh-manoa-covid-19-plan-for-fall/ 

Cameron, K., & Quinn, R. E. (2006). Diagnosing and changing organizational culture: Based on the 
competing values framework (Vol. 59). http://doi.wiley.com/10.1111/j.1744- 
6570.2006.00052_5.x 

Charles Hodges, Moore, S., Lockee, B., Trust, T., & Bond, A. (2020, March 27). The Difference Between 
Emergency Remote Teaching and Online Learning. https://er.educause.edu/articles/2020/3/the- 
difference-between-emergency-remote-teaching-and-online-learning 

Daumiller, M., Rinas, R., Hein, J., Janke, S., Dickhauser, O., & Dresel, M. (2021). Shifting from face-to- 
face to online teaching during COVID-19: The role of university faculty achievement goals for 
attitudes towards this sudden change, and their relevance for burnout/engagement and student 
evaluations of teaching quality. Computers in Human Behavior, 118, 106677. 
https://doi.org/10.1016/j.chb.2020.106677 

Farmer, D. W. (1999). Institutional Improvement and Motivated Faculty: A Case Study. New Directions 
for Teaching and Learning, 1999(78), 87-95. https://doi.org/10.1002/t1.7809 

Fox, K., Bryant, G., Srinivasan, N., Lin, N., & Nguyen, A. (2020). Time for Class - COVID-19 Edition 
Part 2: Planning for a Fall Like No Other. Tyton Partners. https://tytonpartners.com/wp- 
content/uploads/2020/10/10.05.2020.Time-for-Class-CO VID- 19-Part-2-Planning-for-Fall-Like- 
No-Other-V2.pdf 

Galanek, J. D., & Gierdowski, D. C. (2019). ECAR Study of Faculty and Information Technology, 2019 
(p. 29). EDUCAUSE. 

Giovannella, C., & Passarelli, M. (2020). The effects of the Covid-19 pandemic seen through the lens of 
the Italian university teachers and the comparison with school teachers’ perspective. 17. 

Groysberg, B., Lee, J., Price, J., & Cheng, J. Y.-J. (2018, January 1). The leader’s guide to corporate 
culture. Harvard Business Review. https://hbr.org/2018/01/the-culture-factor 

Magda, A. J., & Buban, J. (2018). The state of innovation in higher education: A survey of academic 
administrators (p. 44). The Learning House, Inc. 

McKenzie, L. (2021, April 27). Students Want Online Learning Options Post-Pandemic. Inside Higher 
Ed. https://www.insidehighered.com/news/202 1/04/27/survey-reveals-positive-outlook-online- 
instruction-post-pandemic 

Means, B., & Neisler, J. (2020). Suddenly online: A national survey of undergraduates during the covid- 
19 pandemic. Digital Promise. https://digitalpromise.org/wp- 
content/uploads/2020/07/ELE_CoBrand_DP_ FINAL 3.pdf 

Obendhain, A. M., & Johnson, W. C. (2004). Product and process innovation in service organizations: 
The influence of organizational culture in higher education institutions. Journal of Applied 
Management and Entrepreneurship, 9(3), 91-113. 
https://search.proquest.com/openview/6del 8a0ac7474082 1 cc0f772e8d24656/1 ?pq- 

origsite=gscholar&cbl=25565 


72 


Porter, W. W., & Graham, C. R. (2016). Institutional drivers and barriers to faculty adoption of blended 
learning in higher education. British Journal of Educational Technology, 47(4), 748-762. 
https://doi.org/10.1111/bjet.12269 

Reid, P. (2017). Supporting instructors in overcoming self-efficacy and background barriers to adoption. 
Education and Information Technologies, 22(1), 369-382. https://doi.org/10.1007/s10639-015- 
9449-6 

Rogers, E. M. (2003). Diffusion of innovations, Sth edition. Simon and Schuster. 

Schein, E. H., & Schein, P. (2016). Organization Culture and Leadership. John Wiley & Sons, 
Incorporated. http://ebookcentral.proquest.com/lib/uhm/detail.action?docID=4766585 

Walker, R., Jenkins, M., & Voce, J. (2018). The rhetoric and reality of technology-enhanced learning 
developments in UK higher education: Reflections on recent UCISA research findings (2012— 
2016). Interactive Learning Environments, 26(7), 858-868. 
https://doi.org/10.1080/10494820.2017.1419497 

Zhu, C. (2015). Organisational culture and technology-enhanced innovation in higher education. 
Technology, Pedagogy and Education, 24(1), 65-79. 
https://doi.org/10.1080/1475939X.2013.822414 

Zhu, C., & Engels, N. (2014). Organizational culture and instructional innovations in higher education: 
Perceptions and reactions of teachers and students. Educational Management Administration & 
Leadership, 42(1), 136-158. https://doi.org/10.1177/1741 143213499253 


73 


Culturally and situationally-appropriate professional development design for 
principals of ethnic elementary schools in Yunnan Province, PRC 


DAI Hongwu, 
Assistant Professor of Foreign Languages (on leave), Yunnan Normal University, PRC; PhD 
candidate, organizational leadership, Eastern University, hongwu.dai@eastern.edu 


Dennis Cheek, 
Chief Learning Officer, Values Education Pte. Ltd., Singapore, ststoday@gmail.com; Visiting 
Professor, Innovation and Entrepreneurship, IESEG School of Management, France, 
d.cheek@ieseg. fr 


This paper is part of a set of three papers that together consider a complex learning design case 
emerging in the People’s Republic of China. The physical setting for this endeavor is the 
province of Yunnan in southwestern China, a province of some 48 million persons (Yunnan 
Government, 2019). Yunnan has the largest and most diverse concentration of ethnic groups in 
the nation. 


We have been engaged in determining a suitable learning design for culturally and 
situationally-appropriate professional development for principals of elementary schools in 
Yunnan Province — including incorporating attention to the needs of ethnic minority students. 
Professional development of these principals is undertaken by Yunnan Normal University — 
the higher education institution in the Province of Yunnan charged with the task of preparing 
future teachers and administrators for schools and also for their ongoing professional 
development. Sitting principals are or will be undergoing a systematic program of 
professional development in light of recent changes to compulsory education. This program 
will take about six years to complete across the entire cohort of eligible administrative 
participants. Our work is intended to influence the topics and approaches of some of this 
professional development for these practicing elementary principals as well as adaptations for 
the programs preparing future elementary school administrators and teachers. 


A majority (the exact number is presently unknown) of the 10,688 elementary schools in 
Yunnan are small, rural, ethnic minority-majority schools with 100 or fewer pupils of varying 
ethnicities and ages (Yunnan Provincial Education Department, 2021). These schools are 
spread across a geographic area roughly the size of California, two-thirds of which is 
mountainous and the other third of which lies 1000 feet or more above sea level in high plains 
and river valleys. Principals of these schools face new and unprecedented challenges requiring 
rethinking approaches to professional development and support both for themselves and the 
teachers they lead (Dai and Cheek, 2021). 


A New Cultural Analysis Framework 
Due to a series of interlocking, complex issues that form the context in which professional 


development of elementary school principals in the province must occur, we have pioneered 
application of a new cultural analysis framework we have created tailored to this particular 
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learning design challenge. The starting point for the framework was inspired by the work of 
the French Annales School of historiography in its attempts to write histoire totale — a term 
that suggests that understanding the human past and present requires tools, techniques, and 
perspectives that go well beyond the usual historical considerations of people, places, and 
sociopolitical events placed within a fairly narrow compass of time. Our cultural analysis 
framework utilizes the following five key components we created inspired by the Annales 
approach: 1) geography & climate factors, 2) history & anthropology factors, 3) economy & 
society factors, 4) politics & education administration factors, and 5) management styles & 
learning factors (Dai and Cheek, in press). We believe, as this framework suggests, that the 
business of teaching and learning is always functioning within much wider contexts of politics, 
management, economics, society (both anthropologically and sociologically), climate, and 
geography. Some of these factors change rapidly, while others change quite slowly and almost 
imperceptibly within normal human perspectives. Yet all of these many factors influence what 
happens today and what will happen in future days. 


Space considerations make it impossible to overview all of these cultural analysis components in 
this brief paper. Therefore, we have split the discussion of the factors across three papers with 
factors 1 and 2 discussed mainly in Dai and Cheek (in press), factors 3 and 4 discussed in Dai 
and Cheek (2021), and factors 4 and 5 discussed here — with factor 4 in this paper adding 
material not found in the prior papers. We anticipate writing a longer, more comprehensive 
article, combining substantial attention to all five factors of our cultural analysis framework at a 
future date when the whole picture is considerably clearer than it is at present. 


One important component of our framework is politics and educational administration matters. 
Here we focus on national education law, the ways in which primary and secondary education is 
organized in China, and the implications these matters have for the many roles played by 
principals of elementary schools in Yunnan. 


The Organization of Schools in China 


China’s public education system has been influenced by four factors: 1) ancient cultural heritage 
and learning traditions (especially Confucian thought), 2) leadership literature from the West, 3) 
school management theory and practices from the former Soviet Union, and 4) “leadership tenets 
and principles of the Communist Party of China” (Feng, 2020, p. 3) A four-tiered system of 
educational administration was enacted by the Education Law of the PRC originally promulgated 
in 1995 and whose structural features (detailed in Articles 14 and 15) were left unchanged in its 
2009 and 2015 revisions. At the top is the national Ministry of Education (MOE) in Beijing. 
Below the MOE are the 31 Provincial Education Departments (PEDs), or sometimes called 
Commissions. Below them are the 334 Prefecture-Level-City Education Bureaus (PLCEDs). (A 
prefecture in China is an administrative unit containing several counties; the large cities within 
them have their own educational bureaus.) Finally, underneath them are the 2,850 County 
Education Bureaus or CEBs (Feng, 2020; cf. OECD, 2016). 


The MOE sets all national education policies. The PEDs can establish provincial education 


policies for local variable conditions but they must be congruent with the national education laws 
and policies. The PLCEDs promote and supervise implementation of the provincial education 
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policies. Finally, the CEBs handle the day-to-day operational aspects of local primary and lower 
secondary education. They have the most impact at local levels because they control the funds 
for the nine years of compulsory education and are also where the 253,736 or so primary and 
secondary schools get their support, supervision, and evaluation (Feng, 2020). It is important to 
note that these 250,000+ schools are not the only public schools in China. OECD (2016) reported 
about 514,000 schools below the university level across China including those in autonomous 
regions, special administrative regions, and those under other lines of authority within education; 
attendance in these schools, outside of the first nine years of schooling, are noncompulsory, i.e., 
the higher grade levels up through the end of high school or vocational school are not required 
under compulsory education law. 


Schools throughout China have similar organizational structures which are standardized by 
national education law. Leadership within an individual school resides in four key sets of 
leaders: 1) the School Leadership Team (or School Affairs Committee) — comprised of the 
Principal (and VPs if applicable), the Party Secretary (and associate secretary if applicable), and 
the Chairperson of the School Trade Union; 2) Middle Managers are the Directors of the Office 
of Moral Education, Office for Curriculum and Instruction, and the Office for Scientific 
Research & Teacher Development as well as Ancillary Services; 3) Lower Managers are the 
Heads of Grade Units, Heads for Teaching-Study Groups, and Heads of Lesson Preparation 
Groups; and 4) the Communist Party at the School which is responsible for the work of the 
School Trade Union, Women’s Federation, Communist Youth League (if a secondary school) or 
the Young Pioneers (if a primary school). Since 2016, the Party representative also leads the 
moral education of the school and works closely with the school’s Director of the Office of 
Moral Education to ensure that the Party’s views are consistently promoted to all people within 
the school community (Feng, 2020). 


These various designated leadership roles are, of course, difficult to both fill and fulfill in small 
rural schools — a problem not unlike those faced by rural schools around the world (e.g., Kinkley, 
2019, for US issues and examples). Too few people wearing too many hats for long periods of 
time can be a recipe for declining morale and professional burnout. Many supportive resources 
that are needed in these isolated rural settings are simply not available or if they are made 
available are difficult to access due to the inevitable constraints of time, technology, lack of 
knowledge or skills, or competing demands. Rural schools worldwide pose considerable 
challenges for those responsible for ensuring that education in these schools is consistently of 
good quality and accessible to all who need it (Kong, Hannum, and Postiglione, 2021). Similar to 
many other nations with large numbers of very diverse schools, Chinese education policymakers 
have innovated in a continuing manner to try to address the many challenges faced by school 
leaders. We now describe a number of recent innovations in approaches to educational 
administration within the schools of China. 


The Principal Responsibility System 
A new Principal Responsibility System (PRS) was initially created as part of the 1985 education 
reforms. For the first time, the principal was made responsible for school development planning. 


Given its complexities and the need to prepare principals for this extraordinary expansion in their 
responsibilities, the PRS was phased in over many years as part of a wider overall approach by 
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the government to improve the nation’s schools (Feng, 2020). The PRS is comprised of several 
pillars. First there is the local education authority (county or district education board) who 
supervises the principal. The principal is responsible for leadership of local schools, is the legal 
representative of the school, and under Article 30 of the Education Law is “held responsible for 
teaching and learning activities and administration” (Feng, 2020, p. 27). (It is for this reason that 
we are concentrating our efforts on elementary principals as instructional leaders.) Finally, the 
School Leadership Team chaired by the Principal, is responsible for a wide range of activities 
including various plans, appointments and removals, annual budget, plans for large expenditures, 
appraisal of staff, graduation and enrollment policies, major infrastructure projects, etc. A recent 
change in Party organization at the school level, instituted in 2016, now allows the Principal (if a 
Party member) to simultaneously hold the positions of both Principal and Party Secretary; a 
situation which makes the principal the “paramount leader in his/her school.” While all of this 
may sound very “top down” to those unfamiliar with the education system of China, it needs to 
be noted that across a variety of measures, Chinese schools have been viewed as roughly equally 
autonomous as local publicly-administered schools in Germany, England, and France (Feng. 
2018, 2020). 


Since 2001, primary and secondary schools in many regions have successively established 
school-based management systems that combine government coordination, social participation, 
and independent school management. During this time period, the Ministry of Education sought 
to move all schools away from a knowledge transmission model of learning to increased 
attention on student problem solving, lifelong learning, cooperative learning, and making the 
curriculum more relevant for local situations and contexts. It also incentivized teachers to move 
and remain in poor rural areas through special teacher allowances and subsidies for both primary 
(elementary) and secondary teachers (Kong et al., 2021). These developments have prompted 
changes in the preparation of principals to be instructional leaders as reflected in both textbooks 
for aspiring school administrators in China as well as increased research on the roles and 
responsibilities of principals in serving as instructional leaders (e.g., Wu & Zhou, 2008; Chen, 
2010; Walker et al., 2012; and Walker & Qian, 2020). 


In order to increase accessibility to education, a key component of China’s economic 
development policies, a 2006 revision of China’s compulsory education law (part of Article 2) 
states that “no tuition of miscellaneous fee may be charged in the implementation of compulsory 
education.” Article 6 further clarified that “. . . the State Council and the local people’s 
governments at the county level and above shall reasonably allocate the educational resources, 
promote balanced development of compulsory education, improve the conditions of weak 
schools, take measures to ensure implementation of the compulsory education policy in rural 
areas and areas inhabited by ethnic minority groups, and guarantee that school-aged children and 
adolescents from families with financial difficulties and disabled school-aged children and 
adolescents receive compulsory education.” Article 44 further required in the event of family 
financial inadequacies that “. . . the people’s governments at all levels shall provide them with 
gratis textbooks and give living cost subsidies to boarding students” (Kong, et al.,2021, p. xix). 


While these provisions now clearly establish the government’s priorities, there are clearly still 


many barriers that must be overcome throughout the more rural areas of China and elsewhere. 
One continuing problem, for example, is that children who drop out of compulsory education 
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generally are not pursued in any systematic and continuous manner by the educational system to 
return to school and to complete their studies (Wang, Y., 2014; Bilige & Fan, 2020). It is 
important to acknowledge that dropout problems are in no way unique to China but are a feature 
of educational systems worldwide (cf. OECD, 2021 for OECD countries and Irwin et al., 2021 
for the US). As a consequence, many indicator systems (e.g., OECD) consider schools fully 
enrolled when only 90% or more of the eligible pupils are “enrolled.” 


Since 2001, Chinese primary and secondary schools have tried many largely ineffective changes 
in the process of encouraging further development of their schools and deepening their impacts. 
The main reason for these failures is the lack of systematic thinking among school leaders, 
especially school principals; e.g., they have faced an ever-increasing set of leadership 
responsibilities over the past three decades and they may lack ability to think conceptually about 
the many complex problems that now confront them (Liu, 2013). We suspect that there are a 
multitude of reasons beyond just the lack of systematic thinking among school leaders, especially 
in semi-isolated and geographically challenging areas of China such as Yunnan (cf. Wu, 2020). 


Five years ago, in an attempt to deepen reforms and improve results of primary and secondary 
schooling across China, the Organization Department of the CCCPC and the Ministry of 
Education (2017) promulgated new provisional “measures for the management of leaders of 
primary and secondary schools.” It complemented and further clarified some earlier guidance 
issued by the Ministry of Education (2013). The provisional measures deal with a series of 
matters including: 1) entry requirements and professional qualifications for principals and other 
administrators, 2) rules and procedures for selection, appointment, and service term setting (e.g., 
serving a maximum of 12 years as a principal in the same school), 3) accountability and 
performance appraisal, 4) establishment of a career path and motivation system, 5) supervision 
and restraint mechanisms (to appropriately limit principal’s powers), and 6) termination or 
dismissal mechanisms for school leaders. The promulgation of these rules simultaneously 
established the primacy of principals and also delineated for the first times the limits of their 
authority and powers. 


Educational Administration Challenges in Yunnan 


About eighty percent of Yunnan elementary schools are found in rural areas where 25 formally- 
recognized ethnic groups are heavily concentrated. These groups move freely across large areas, 
including both adjacent countries and other regions of China. Schools in these regions are often 
ethnic minority-majority with 14+ indigenous languages spoken by children and their parents, in 
addition to parents and children speaking, reading, and writing Mandarin with varying degrees of 
fluency and proficiency (Chinese Rural Education Development Research Institute, 2017; Dai & 
Cheek, 2021). Most schools are small (about 100 students) as we have noted, and many are 
residential Monday-Friday due to sparse and widely dispersed populations across these 
mountainous regions (Lei & Zhang, 2014). The number of boarding schools is slowly but 
steadily increasing as authorities continue to systematically decrease the number of total schools 
due to overall declining numbers and the provision of subsidies for families to board their 
children at the schools. China’s efforts on consolidations appear to be decreasing the dropout rate 
as well; perhaps larger numbers both increase school activities and provide more opportunities 
for students to form friendships and inhibit their dropping out (Yu, 2013).Another new 
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phenomenon are children who remain at school through the weekends and thus board there 
throughout the school year. Often these are students whose parents are away working in the cities 
with no local relatives to care for them (Wang, Dong & Mao, 2017). Student populations in the 
vast rurality of Yunnan tend to vary across the year due to poverty, isolation, hazardous journeys 
to school, annual migratory patterns, parental need for their free labor, and health (Dai & Cheek, 
2021). 


The majority of teachers and administrators in these ethnic minority-majority schools are part of 
the ninety-two percent Han culture of the PRC; a group which generally agrees with the view 
frequently expressed that literacy in China refers to competencies in Mandarin, not in other 
dialects or languages (Yamada, 2021).There has been mixed success in China with approaches to 
fully bilingual or trilingual approaches to language learning and use (Tsung, 2009, 2014; Zhang, 
Y. J., 2013). Valuable programs include plenty of opportunities for immersion in a language, 
extensive discourse in the language that is related to the school’s regular curriculum and 
highlighting practical uses and applications for the language in question. Many successful 
programs can be found in cities but in rural areas the many challenges that must be overcome 
often overwhelm well-intentioned efforts (Tsung, 2009, Sude and Dervin, 2020). 


Recent national pronouncements and actions regarding unrest in the borderlands of China will 
undoubtedly result in changes to government policies regarding local ethnic language learning 
within the school curriculum and change how schools approach issues of citizenship 
development and holistic development of children (e.g., Xi, 2014, 2017). The exact nature of 
these changes is currently unknown. 


The need for adaptive leadership has perhaps never been greater. Principals need to be 
instructional and institutional leaders who can engage in critical thinking about the socio- 
technical system called formal public education as it faces accelerated change and increasing 
demands (Blasé et al., 2010; Kinkley, 2019). Principals need to become more culturally attuned 
to the needs of their teachers, pupils, and communities in areas of the country where the school is 
vital to community-wide continuity, cohesion, and continuing social and economic development 
(Cherng et al., 2019; cf. Pasanchay, 2019). Principals need time and opportunity to identify, 
explore, and question their own beliefs and understandings about the children and families of 
local ethnic groups. This includes viewing the sociocultural dimensions of students as a resource 
strength for students’ ongoing cognitive, social, emotional, and citizenship development within a 
holistic frame of reference (cf. Yang et al., 2021). At the same time, principals are responsible 
within their schools for achieving the government’s priority for Mandarin to be the common 
language (“Putonghua’’) of the nation (Yamada, 2021). 


There is much about our engagement we do not presently know (e.g., Amzat, 2019). Relevant 
research is meager regarding many relevant issues (Su et al., 2019). There are, however, 
interesting pointers from ongoing educational and psychological research across an array of 
fronts including: reducing socioeconomic disparities (Destin, 2020), honing reasoning ability 
(Bunge & Leib, 2020), improving children’s multiple language skills (Hulme et al., 2020), the 
physical context of child development (Evans, 2021), taking social, emotional, and behavioral 
skills seriously (Soto et al., 2021), promoting student self-regulation and transfer (McDaniel & 
Einstein, 2020), addressing discrimination (e.g., Bettache, 2020), understanding 
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neurodevelopmental effects of childhood adversity (Smith and Pollak, 2021), and providing 
supportive environments that enhance teachers’ self-efficacy (Lackey, 2019). 


In addition to socio-technical systems thinking about the educational system as a whole 
(McWalters & Cheek, 2000), principals also need a formal, flexible, and relationship-focused 
way to apprehend the need for changes and ways to work through the worldviews, means, 
methods, and processes that will position their teachers, schools, and communities to 
successfully respond to emerging challenges. We suspect that Soft Systems Methodology (SSM) 
will be a useful tool to employ with principals as part of their own professional development and 
to develop comfort and skills using SSM with teachers and community leaders within their own 
respective school service areas (Jackson, 2019). 


Conclusion 


The challenge is to create a working model of how to integrate various factors into a flexible 
approach that will place principals at the center of a systematic learning and action research 
program that will produce culturally and situationally appropriate and educationally effective 
changes to learning on the part of both rural children and the adults who work with them. 
Success at such a venture will greatly affect the future of these rural children as citizens of the 
People’s Republic of China. 
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Social media can serve as useful learning tools for teenagers. The term social media first 
emerged in tandem with Tim O’Reilly’s declaration of Web 2.0 as a participatory web (Hogan & 
Melville, 2015). Now, a wide range of online activities with multiple contributors are considered 
as social media, ranging “from collaborative encyclopedias such as Wikipedia, to social network 
sites (SNSs) Facebook and Twitter, photo-sharing sites Instagram, and social news site Reddit” 
(Hogan & Melville, 2015, p. 421). As social media is pervasive in today’s teenage life, there are 
concerns regarding the harmful impacts of social media on teens. However, we, educational 
technologists, need to proactively find ways to use social media to improve learning and prevent 
any possible side effects. To fully utilize the new educational opportunities afforded by social 
media, we should examine the roles of social media in education, as well as the benefits and 
challenges it presents (Greenhow et al., 2019a, b). 

Recent studies found that one of the most common informal learning behaviors among 
teens on social media is future/college planning (Bagdy et al., 2018; Rutledge et al., 2019). The 
researchers found that teens use social media to seek out college information by following the 
institutional accounts of potential colleges as well as by connecting with college students. Given 
the lack of research on adolescent learning with social media (Greenhow & Lewin, 2016), it is 
important to synthesize the research to date and figure out the path for the future study. 

Thus, this review aims to (a) synthesize the literature on teenagers’ use of social media to 
learn about college and (b) to offer educational practitioners and researchers suggestions for 
supporting college-bound teenagers. The research questions of the study include: 

1. What are some features of previous literature exploring the topic? 
2. What social media affordances do teens use to learn about college? 
3. What are some obstacles teenagers face while learning about college on social media? 


Method 


Search and Selection Procedures 

I searched the databases ERIC ProQuest, ScienceDirect, and Education Full Text from 
August 27" to October 7" in 2020 to find the literature. The second search was conducted on 
August 6" in 2021 to ensure that the recently published studies are included. I consulted with a 
school librarian to find databases and search terms. The search terms used were (teen OR 
adolescent OR youth OR “high school student”) AND (“college access”) AND (online OR 
internet OR “social media” OR “social network”). I limited the results to be peer reviewed 
scholarly journals and be written in English. I did not use any specific limit in publication year, 
considering that social media is a relatively recent phenomenon. 

The search yielded a total of 641 articles from the databases. After removing 12 
duplicates, I first reviewed the articles’ titles based on the inclusion and exclusion criteria, 


85 


resulting in 386 articles for further review. The abstract review yielded a total of 96 articles. 
After reviewing the full text of the articles, I found five that were eligible. Additionally, I looked 
through the reference lists in the selected articles to find relevant literature. I also utilized Google 
Scholar's "relevant articles" and "cited by" features to find more recent relevant publications. 
These searches yielded three more articles, bringing the total to eight. 


Selection Criteria 

I included the literature examining both teenagers and their learning about college 
through social media. Only empirical studies published in peer-reviewed journals written in 
English were considered as eligible literature. I excluded the literature on the use of information 
and communication technologies in general without providing any data regarding social media. 
In addition, I excluded the research on college students’ social media use for college adjustment. 
I also excluded studies whose participants’ demographic information was not provided in detail. 
As a result, a total of eight articles met the criteria for inclusion in the literature review. 


Results 
RQ 1: Study Characteristics 


For the eight reviewed studies, I identified some of their key features, including 
participants, theoretical framework, social media sites examined, and data collected (Table 1). 


Table 1 
Features of the Literature Reviewed 
Study Framework Methods, Data Collection, and Sample Social Media 
Sites 
Brown et al. Information Qualitative Not specified; 
(2016) literacy Interviews Online in 
Students from six high schools in Michigan general 
(N = 68) 
Ellison etal., Social Qualitative Not specified 
(2014) capital Interviews 
Students from three high schools in Michigan 
(N = 43) 
Marciano New media Qualitative Facebook 
(2015) literacy Interviews, focus groups 


Black and Latina first-gen college applicants 
from a single high school in New York (N = 


10) 
Marciano New media Qualitative Not specified 
(2017) literacy Interviews, focus groups 


12" grade Black and Latina first-gen college 

applicants from a single high school in New 

York (N = 10) 
Greenhow & Social Mixed methods Not specified 
Burton (2011) capital 
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Survey: Students from low-income families 
who were recruited from three urban high 
schools in the upper Midwest (N = 607) 
Interviews: Participants selected from those 
surveyed (N = 11) 


Greenhow & New Qualitative MySpace 
Robelia literacies Interviews, talk-alouds, content analysis of 
(2009) MySpace profiles 


High school students from low-income 
families in the upper Midwest (N = 11) 


Wohn et al. Social Quantitative Facebook 
(2013) capital Survey 

High school students 

(N = 789) 

Compared first-gen and non first-gen students 
Rutledge et al. Not Qualitative Not specified 
(2019) specified Survey and activities: 


High school students from a single high 
school in Florida (N = 48) 

Interviews: 

High school students from the same school (N 
= 37), 

Administrators and teachers from the same 
school (N = 18) 


Note. First-gen = First-generation 


Participants of the Studies 

All eight studies reviewed used a sample consisting of high school students. Five studies 
considered participants’ first-generation status by purposefully including them in the samples 
(Brown et al., 2016; Ellison et al., 2014; Marciano, 2015, 2017; Wohn et al. 2013). Notably, 
Wohn et al. (2013) compared first-generation students and non first-generation students in terms 
of social capital and college aspirations. All of the studies except for Rutledge et al. (2019) stated 
that they included students from low-income households. Regarding the locations of the 
participating schools, four of the studies sampled participants only from urban high schools 
(Marciano, 2015, 2017; Greenhow & Burton, 2011; Greenhow & Robelia, 2009); other three 
studies had participants both from urban and rural areas (Brown et al., 2016; Ellison et al., 2014; 
Wohn et al., 2013). Marciano (2015, 2017) and Rutledge et al. (2019) recruited students from a 
single high school in New York and a single high school in Florida, respectively. 


Theoretical Frameworks 

Three of the reviewed studies adopted the social capital framework to situate their 
research. Social capital refers to the ability of individuals to access and deploy resources in their 
social network (Wohn et al., 2013). Ellison et al. (2014) examined teens’ online and offline 
experiences associated with two different forms of social capital: bridging and bonding. They 
discussed how teens’ experiences related to these two types of social capital reshaped their future 
aspirations. Greenhow and Burton (2011) emphasized the significant role of social capital in 
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education and psychological well-being. They examined how social media use of high school 
students from low-income families related to their social capital. Wohn et al. (2013) examined 
how social capital accrued through parents, peers, and Facebook Friends can affect teens’ college 
application efficacy and expectation of college success. 

Four of the reviewed studies adopted the literacy perspective. While they used different 
terms like information literacy, new literacy, and new media literacy, they all focused on teens’ 
communication practices on social media. Brown et al. (2016) investigated teens’ college 
information seeking practices online to understand what skills and strategies they use to access 
information about college and how they analyze and evaluate the information found online. 
Greenhow and Robelia (2009) emphasized the importance of online communication, especially 
for low-income students, because it allows them to interact with peers based on their specific 
interests rather than geography. They looked at how teens from low-income families use 
MySpace from the perspective of new literacy. Marciano (2015, 2017) explored the new media 
literacy practices of Black and Latina/o youth attending urban public high school. 


Research Designs 

One study used only quantitative data. Wohn et al. (2013) used questionnaires to inquire 
about their college aspirations and different types of social capital they had (i.e., demographic, 
structural, immediate network, extended network). They conducted hierarchical linear regression 
analyses to identify the added variance due to the inclusion of the four social capital factors. 

Six research only relied on qualitative data. Brown et al. (2016) conducted in-depth 
interviews to examine teens’ online practices to learn about college. Ellison et al. (2014) also 
employed in-depth interviews to investigate how teens’ online and offline experiences shaped 
their understandings of possible life paths. Marciano (2015, 2017) used interviews and focus- 
group interviews. They adopted a social participatory youth co-researcher approach, in which the 
researcher recruited focal participants, and the focal participants invited peer participants to the 
study and performed interviews as coresearchers. Greenhow and Robelia (2009) conducted 
qualitative case studies to explore teens’ novel practices on MySpace by using interviews, talk- 
alouds, and content analysis of MySpace profiles. Rutledge et al. (2019) used a multilevel 
exploratory case study. They taught social media lessons to 48 high school students in the first 
study year and 37 students in the second. Data were collected through survey, audio-records of 
the lessons, field notes, activity worksheets, and interviews. They also interviewed 17 faculty 
members and administrators. 

One study used both quantitative and qualitative data. Greenhow and Burton (2011) first 
conducted a survey with high school students from low-income families and used multiple 
regression analyses to see how intensity of their use of social media could predict their social 
capital. They selected 11 participants from those surveyed and carried out semi-structured 
interviews in order to complement the survey data. 


Social Media Sites 

Two of the studies indicated their focus on teens' use of Facebook (Marciano, 2015; 
Wohn et al., 2013), and one on MySpace (Greenhow & Robelia, 2009). The other five studies 
looked at teens' overall use of social media rather than specific social media platforms. Several 
sites, including Twitter, Xbox, and YouTube, were mentioned by the research participants and 
were covered in those publications. 
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RQ 2: Social Media Affordances to Learn About College 
The reviewed studies show that teens use social media in various ways to learn about 
college. I identified the five social media affordances teens use: vicarious experience, 
information seeking, identity development, emotional sharing, and schoolwork support. 


Vicarious Experience 

Many of the reviewed literature suggest the promising role of social media in exposing 
teens to a broader range of people. For instance, Brown et al. (2016) found that teens were 
monitoring social media profiles of college students at their "dream schools," which informed 
them about college life and functioned as a motivator. Furthermore, they revealed that teens on 
social media would inadvertently encounter college-related information from their peers and 
family members in college. Seeing their college lives shared on social media helped teens make 
school decisions and inspired them to research more about college. 


Information Seeking 

Wohn et al. (2013) revealed that having Facebook friends who could give college 
information and advice was a positive predictor for the first-generation students’ expectations 
about college success. Ellison et al. (2014) reported that a teen who was a member of a special 
interest Facebook group discussed attending college out of state with the group members. 
Greenhow and Burton (2011) described a teen encountering a college student going to her 
interested school on MySpace. The teen exploited the opportunity by inquiring how to get into 
the school and their personal opinions about the school. Rutledge et al. (2019) also noted that 
teens networked with college students they encountered on the college’s Facebook page and 
messaged them to ask about school information. Greenhow and Robelia (2009) suggested that 
first-generation students got advice on their college application by contacting their former 
classmates. Similarly, Brown et al. (2016) reported that teens used social media to ask college- 
related questions to their peers and cousins with college experience. 

The advanced technological features facilitated disadvantaged teens’ information 
seeking. For example, Marciano (2015) described a teen from an immigrant household who 
could not afford college tours. Thanks to a friend’s suggestion, the teen was able to take a virtual 
tour through YouTube videos and Facebook pages, which the teen found very beneficial. 
Moreover, Rutledge et al. (2019) reported that teens were following the colleges’ official 
Instagram, Facebook, and Twitter accounts, where they read the posts and comments and interact 
with college students. The reviewed studies have also reported teens using the tag and hashtag 
features to learn about college (Brown et al., 2016; Marciano, 2015). Brown et al. (2016), for 
example, found that teens utilize hashtags to locate the social media posts about a university’s 
marching band and Historically Black Colleges and Universities. 


Identity Development 

Some of the reviewed studies show that teens used social media to share their college 
aspirations and achievements. According to Brown et al. (2016), some teens updated their 
Facebook status to share their college plans. Friends and family members showed interest in their 
college application process and gave them encouragement to keep going. Some students posted 
their admission letters on social media. Similarly, Greenhow and Burton (2011) reported that 
teens shared their educational plans on MySpace profiles. Sharing their college aspirations and 
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attainment on social media enabled teens to build and maintain college-going identities and 
recognize peers who had similar plans. The social support they received reinforced such identity. 


Emotional Sharing 

Social media became a place where teens vented their stress about college planning. 
Greenhow and Robelia (2009) found that when teens expressed their concerns about college 
planning, their former classmates, now in college, gave emotional support by leaving supportive 
comments on their MySpace accounts. 


Schoolwork Support 

Marciano (2015, 2017) highlight how teens utilize social media to help one another meet 
academic requirements for college readiness. They found that teens discussed schoolwork 
through text messages and Facebook posts. Greenhow and Robelia (2009) and Rutledge et al. 
(2019) also suggest how teens use social media to support schoolwork, by asking questions about 
deadlines, organizing study groups, exchanging educational materials, and collaborating on 
homework. Furthermore, Marciano (2015) described that a teen posted her poem on Facebook 
that could be possibly used for her college application essay. She tagged a well-known literacy 
organization in her post, and one of the organization's mentors gave critical feedback on her 
writing. The teen purposely utilized the tag function to attract the organization's attention and 
was successful in gaining comments to improve her work. 


RQ 3: Obstacles Teens Face While Learning About College on Social Media 
Despite the advantages of social media identified above, the reviewed studies have also 
noted some obstacles that teenagers could face while using social media to learn about college. 
In this section, four major challenges will be discussed. 


Difficulties with Applying Broad Information to Their Specific Contexts 

Brown et al. (2016) revealed that while teens reported being able to identify and access 
college information online, they struggled to evaluate and apply the information to their specific 
needs. The participants described a large amount of information on the web as overwhelming 
due to its lack of meaningful context. The researchers thus emphasize the importance of 
knowledgeable translators who can help them understand the information found online. The term 
“knowledgeable translator” is defined as “social contacts that students access(ed), via online and 
offline channels, in order to make sense of generic information and apply it to their specific 
informational need” (Brown et al., 2016, p.110). The researchers also discovered that a few 
participants considered social media as an unreliable source of information and preferred 
authoritative sources such as guidance counselors. However, at the same time, social media may 
help teens make sense of complex information. For instance, some participants described how a 
guidance counselor at their school curated information for a Facebook group by filtering 
information relevant to students. The counselor also answered students’ questions about online 
information and directed them to additional resources. Teens also reached out to peers and 
family members who were current undergraduates or college graduates, to contextualize 
information about college. In sum, while teens may have difficulty interpreting information 
obtained on social media, it could also be a place for them to find knowledgeable translators. 
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Potential Negative Influences of Social Media 

Wohn et al. (2013) revealed in their quantitative research that teens’ frequent use of 
Facebook was negatively related to their expectations of college success. This result implies that 
not all activities on social media might be beneficial for teens’ college aspirations. Moreover, 
Wohn et al.’s (2013) found that for non-first generation students, the number of Facebook friends 
were negatively related to their college application efficacy. For first-generation students, the 
emotional support from Facebook friends was a negative predictor for their college application 
efficacy. Though the study does not provide further evidence to clarify such relationships, the 
findings indicate that Facebook friends and their emotional support might have negative impact 
on college application efficacy. 


The Nature of Social Media Platforms That May Limit Network Expansions 

Social media platform’s distinctive features could affect teens’ learning about college. 
For instance, Ellison et al. (2014) found that on Facebook, teens tend to friend their schoolmates 
and family members rather than reshaping their networks. The participants mentioned the 
“normative pressure to indiscriminately friend schoolmates” (p. 526) on Facebook. Such a norm 
had teens have little control over their friending decisions and thus kept them from being 
exposed to various worldviews or novel information. 

However, the same culture on Facebook where teens network with their existing 
relationships appears to be beneficial in terms of getting social help. Specifically, Marciano 
(2015, 2017) found that when teens post questions about school assignments on Facebook, they 
were able to get quick, multiple answers from their peers enrolled in the same classes. Greenhow 
and Robelia (2009) and Greenhow and Burton (2011) also found similar results on MySpace, 
where teens asked for help with their school tasks and received needed help. 


Discussion 


RQ 1: Study Characteristics and Suggestions for Future Research 

All the reviewed studies recruited high school students as participants, and the majority 
of them considered the participants' socioeconomic backgrounds. They used literacy and social 
capital as their primary frameworks to explore the topic. Empirical studies based on a broader 
range of theoretical frameworks may help us to see teens’ social media use from diverse angles 
and thus enrich the research conversation on this topic. Potential frameworks that could be 
employed include communities of practice, self-efficacy, affinity spaces, cultural historical 
activity theory, and self-regulated learning theory (Greenhow et al., 2019b). In terms of research 
methods, one study only used quantitative data to identify the connections between variables, 
whilst six only used qualitative data. One study employed both quantitative and qualitative data. 
Recent social media research has begun to be more methodologically varied, by adopting, for 
example, big data techniques and social network analysis (Greenhow et al., 2019b). Future 
researchers could investigate such opportunities and utilize different methods to achieve their 
research goals. When it comes to social media sites, three studies were primarily focused on 
Facebook and MySpace while others did not identify any specific platforms and instead looked 
at teens’ social media use in general. Though either approach could be used depending on 
research goals, focusing on specific tools may allow us a more nuanced understanding of 
different tools (Orben, 2020). 
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RQ 2: Suggestions for Leveraging Social Media Affordances 

The current review identified five social media affordances that can be used to learn 
about college, indicating that we can use them to better help college-bound teenagers. First, we 
can support students’ technical access to social media by reducing the regulation on the use of 
phones and social media during the school day. Blocking social media sites at school may 
separate teens from potentially valuable college information (Brown et al., 2016; Marciano, 
2015). According to Enriquez (2011), undocumented immigrant Latinx students found weak ties 
with other undocumented students to be more useful than teachers, counselors, and school 
officers in meeting their unique information needs. Teens can possibly use social media to 
connect with potential weak ties who may provide information tailored to their own situations 
that they may not be able to obtain at school. 

Second, we could integrate social media into curricular activities to benefit from its 
learning affordances (as suggested by Marciano, 2015). Previous scholars have emphasized the 
importance and benefits of bridging formal and informal learning contexts with the use of social 
media attributes (Greenhow & Askari, 2017; Greenhow & Lewin, 2016). For example, we could 
encourage students to share classroom artifacts via social media by tagging experts and peers to 
receive help with their work. This way, we can utilize the social media affordances to support 
schoolwork and have our students associate social media as a learning tool. 

Third, we should also help teens improve new media literacy skills to fully exploit the 
benefits of social media (Brown et al., 2016; Marciano, 2015). New media literacies include 
traditional literacy such as reading and writing, research skills, technical skills, critical analysis 
skills, but more importantly, “social skills developed through collaboration and networking” 
(Jenkins et al., 2009, p. 29). The new media literacy will be crucial in finding credible online 
resources and making more informed college-related decisions. 

Fourth, we could design and develop an intervention for teens’ college preparation by 
using social media affordances. Though many teens use social media on a daily basis, they might 
not be fully aware of its affordances to help their college-going process. We could let them know 
the availability of institutions’ social media accounts and potential uses of social media to reach 
out to people attending their dream schools. We could also teach skills to find specific 
information about colleges using their appropriate hashtags. Such endeavors may involve design- 
based research aimed at developing an intervention to improve teens’ informal learning about 
college using social media (Greenhow et al., 2019b). 


RQ 3: Suggestions for Overcoming Potential Obstacles 

Despite all the merits of social media mentioned earlier, this review also highlighted 
some obstacles teens experience while learning about college through social media. Based on 
them, I give specific suggestions for practitioners and future researchers. First, educators should 
provide teens with knowledgeable translators who can help teens make sense of information 
found online and apply it to their specific needs. Educators themselves could serve as translators 
for teens at school or district levels. In addition to face-to-face assistance, we could also consider 
utilizing social media. For instance, school guidance counselors could create a Facebook group 
page or an Instagram account where they provide college-related information tailored to their 
students as suggested by Brown et al. (2016). Considering that disadvantaged students are less 
likely to seek help from guidance counselors (Holland, 2015), using social media as a means to 
communicate college information could be a possible strategy for making college information 
more accessible to a larger student population. 
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Second, researchers should examine how specific social media activities have different 
impacts on teens’ college aspirations. Wohn et al. (2013) found a negative association between 
teens’ frequent use of Facebook and their college aspirations. The frequency indicator may not 
be sufficient to draw upon to explore how social media usage impacts teens. It is important to 
stop using the blanket term "social media use" and instead focus on specific social media 
practices. Future studies should examine what types of social media activities are associated with 
teens’ college aspirations and with their actual college access. 

Third, future studies could investigate the impacts of social media friends. The reviewed 
studies suggested mixed results regarding how social media friends and their emotional support 
affect teens’ college aspirations, which implies the complicated nature of peer influence. Other 
previous studies have examined how peers on social media can possibly affect teenagers through 
fear of missing out (Marengo et al., 2021), peer comparison (Chua & Chang, 2016), and drama 
(Dennen et al., 2018; Rutledge et al., 2019). Such complex phenomena could complicate how 
social media friends influence teens in terms of their college aspirations. 

Fourth, researchers should examine different social media platforms’ distinctive features 
and their impacts on teens’ learning about college. This review suggests that a social media norm 
could be a double-edged sword depending on how teens use it. Future research might look into 
teens' learning practices on various social media sites, as well as their potential advantages and 
drawbacks. For instance, Dennen et al. (2020) explored the role of six prominent social sites in a 
high school environment (i.e., Instagram, Snapchat, Facebook, Twitter, YouTube, and Pinterest). 
Alhabash and Ma (2017) investigated college students’ use of different platforms and nine 
distinct motivations. Similar to them, we could examine how teens learn about college through 
various social media platforms and further explore best practices for each platform. 


Conclusions 


This study reviewed the eight literature investigating teens’ use of social media to learn 
about college. Theoretically, current studies on this topic have been primarily situated in literacy 
theory and social capital theory. Methodologically, the current studies leaned more toward the 
qualitative approach. The review suggests five social media affordances in learning about 
college: vicarious experience, information seeking, identity development, emotional sharing, and 
schoolwork support. On the other hand, there were some potential barriers teens may encounter 
when learning about college on social media. First, they were having difficulty understanding 
generic information obtained online. Second, some social media activities may be detrimental to 
teens' college aspirations. Third, the norms of social media sites may impede teens from learning 
about college. This research has implications for how to use social media affordances to better 
help college-bound teens in their college learning. 
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Abstract 


Note-taking is an effective learning strategy for SRL development. However, due to the lack of 


opportunities to learn note-taking, learners cannot write complete notes. Therefore, from the 


perspective of co-regulated learning (Co-RL), we proposed that learners learn note-taking from 


each other. This study aimed to evaluate NoTAS, a note-taking support system using tablet 


devices. We asked 33 participants in the experiment to evaluate whether NoTAS promotes Co- 
RL and note-taking. The questionnaire results showed that the learners using NoTAS can learn 
and take notes through interaction with others. However, there were some cases where NoTAS 


prevented learners from note-taking, suggesting the need for improvement. 


Keywords: Note-taking, Co-RL, Nudge theory, CSCL, Learning visualization 
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1. Introduction 


1.1. Self-Regulated Learning 


OECD (2018) indicated that learners need to develop self-regulated learning (SRL) 
competencies in the Learning Compass, a vision for the future of education. SRL is the ability to 
cycle through forethought, performance, and reflection on their own to learn effectively (Usher 
and Schunk, 2018). In the field of educational technology, much of the research on SRL 
development has focused on outside class learning support, such as encouraging reflection and 
learning management using assignments, portfolios, and learning management systems. However, 
outside class learning support can be burdensome in terms of preparations by teachers and 
activities by learners. On the other hand, Nilson (2013) proposed note-taking as one of the 
effective learning strategies for SRL development for in-class learning support. 


1.2. Note-Taking and Support Methods 


Note-taking has two types of features: encoding and storage functions (DiVesta and Gray, 
1972). The encoding function facilitates recognition processing by combining the learning 
contents with the learner's prior knowledge through writing notes. The storage function enables 
effective review by reading notes. Morehead et al. (2019) suggest that many learners take notes 
in class but cannot write complete notes because they have few opportunities to learn note-taking 
strategies. One of the ways to support note-taking is to distribute class material. Class material 
clarifies the main points of the lesson (Kiewra, 1989). Furthermore, writing directly on the class 
material facilitates understanding of the class (Avval et al., 2013). However, only a few learners 
take organized notes on the teacher’s explanations. Therefore, Lannoe and Miller (2019) suggest 
that more support is needed to encourage more learners to take notes. 

Another practical support strategy is to provide feedback on note-taking. For example, 
Beaudoin and Winne (2009) developed the “nStudy system,” which allows teachers and other 
learners to provide comments and detailed feedback on learners’ essays. While “nStudy” can 
provide detailed feedback, it is difficult for instructors to provide note-taking instruction 
constantly, and we do not have enough time to share and discuss notes in the class (Nilson, 2013). 


1.3. Who Provides Feedback on Note-Taking? 


Hadwin et al. (2017) proposed Co-Regulated Learning (Co-RL) as a recent trend in 
research on regulated learning. Co-RL is a learning to regulate one's learning through interaction 
with others. Therefore, we suggest that note-taking feedback among learners is possible by 
applying the Co-RL theory. 

In Japan, the Ministry of Education, Culture, Sports, Science, and Technology (MEXT, 
2020) has promoted a policy to allow all learners to own devices to develop networks. Thus, 
many countries witnessed the growing trend of teachers and learners using Information 
Technology (IT) in classes in recent years. An example of such a feedback system is the 
“Metaboard,” a learning analytics dashboard that supports learners’ metacognition and SRL by 
visualizing their learning behavior (Chen et al., 2020). Furthermore, learners can take notes using 
a pen and tablet in class without any stress (Ozcakmak and Sarigéz, 2019). Therefore, learners 
can use tablets for longhand note-taking instead of paper and keyboard. However, there is little 
study on supporting longhand note-taking with tablet devices applying Co-RL theory. 
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We hypothesize that learners who cannot take notes would be aided by assuming that the 
notes that many learners were taking were correct and providing them with feedback using 
tablets. Thus, to promote interaction among learners, we focus on “Nudge theory,” which has 
been studied extensively in the field of behavioral economics in recent years (e.g., Thaler and 
Sunstein, 2009). 


1.4. Nudge Theory in Education 


Thaler and Sunstein define the nudge as any element of choice behavior that changes 
people’s behavior predictably without narrowing the choice or significantly changing the 
economic stimulus (Thaler and Sunstein, 2009). Research on nudge has also been applied to 
educational studies (Weijers et al., 2020). However, most research is confined to nudging on 
teaching policies, with little research on nudging the learning strategies among learners in class. 
Here, we define the educational nudge as improving one’s note-taking by referring to the colors 
and positions of others’ note-taking. In this research, we aim to learn each others’ learning 
strategies, such as note-taking in the class. 


2. Purpose 


In this study, we evaluated NoTAS developed by Kondo et al. (2021) that uses nudge to 
provide feedback on note-taking among learners in class. We applied NoTAS to subjects and 
evaluated its effectiveness by using questionnaires. There are two perspectives for the evaluation. 

1. NoTAS promotes interaction with other learners (Co-RL) in the class. 

2. NoTAS promotes note-taking based on the others' note-taking. 


3. Methods 
3.1. System Overview 


Nudge for Note Taking Assist System (NoTAS) is a note-taking feedback system that can 
be accessed by up to 45 people at a time, excluding teachers (Kondo et al., 2021). NoTAS is 
available on web browsers such as Safari and Google Chrome. NoTAS has three functions: note- 
taking function, learning log function, and visualization function. Learners can use the note- 
taking function to write notes and highlights text directly on class material displayed on their 
tablet using their tablet pen. Two types of markers are available for learners to highlight the 
important parts and the unclear parts. The learning log function has two types: log collection and 
log confirmation. The log collection allows learners to save their note-taking sequentially as they 
write and erase notes and highlights in class material. Teachers can view the log of the learners' 
note-taking processes by specifying the learner (User ID) and the material (Class ID) using the 
log confirmation. Furthermore, the learning log function uses BASIC authentication so that only 
certain learners and teachers can access the system. The visualization function is the main of 
NoTAS. When a learner writes notes and highlights on the class material with visualization 
function, the approximate location of the notes and highlights written by others on the class 
material is visualized on the same material in almost real-time. 

The interface of NoTAS consists of four layers: note-taking layer, learning log layer, 
learning visualization layer, and class material layer. On NoTAS, the learner directly writes on 
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the material that has been distributed via the tablet device. The note-taking layer and learning log 
layer correspond to the note-taking function. Then, to improve note-taking among learners, 
NoTAS collects and visualizes information, such as the place and time of writing of all learners, 
which facilitates learning. The learning visualization layer shows the position and timing of other 
learners’ note-taking. Since this layer overlaps with the number of learners in class, the more 
learners fill in the same part, the darker the color becomes. As a result, the areas written by more 
learners are emphasized. Figurel shows an example of the visualization function of NoTAS. 
Each color has the following meanings. 

@ The red areas mean that other learners wrote notes. 

@ The yellow areas mean that other learners highlighted important parts. 

@ The blue areas mean that other learners highlighted unclear parts. 


> 
25% of the class p55. rN a 
highlighted the @ 1 SOMES 220 NRA BCT = 
unclear parts. WRAMS RE BE, Btec SaRE 
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Figure 1. Visualization interface 
3.2. Procedure of Research 


We recruited university students to evaluate the NoTAS. The participants in this research 
were 33 students (males: 22, females: 11). The average age of the participants was 22.7 years old. 
We distributed a tablet (iPad 6th) and a tablet pen to the participants. Before the class, the 
participants had an opportunity to practice the operation of NoTAS. The experimental group (n = 
15) took the class using the visualization function of NoTAS. On the other hand, the control 
group (n = 18) took the class without using the visualization function of NoTAS. The class 
content was four instructional design theories, and the participants watched the class video using 
a projector. Figure 2 shows the procedure of research. 

In addition to the participants, five collaborators wrote the contents set by the first author 
on the class material at a set time. This operation was conducted to verify the effect of the 
visualization function. We asked the participants on the face sheet, “If they had ever taken a class 
using a tablet and tablet pen,” and found no significant difference between the two groups, @(1) 
= 0.02, p = .88, Z= .03. 
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Experimental group Control group 


Face sheet & Pre-test 
| 
Explanation of system 


Visualization Non-Visualization 


Students take the class |} Students take the class 
using the visualization || without using the 
function. visualization function. 


Post-test & Questionnaire 


Figure 2. Procedure of research 


3.3. Questionnaire 
In this research, we surveyed questionnaires from three perspectives as follows: 
Community Awareness 


We quoted 15 items related to community awareness for Classroom Community Scale 
(Rovai, 2002). We have partially rewritten the text to be more consistent with the purpose of this 
study. Moreover, we created and added 3 original items about others’ note-taking. We asked all 
items using a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). 


Social Presence 


We adopted 3 items proposed by Short et al. (1976) to measure social presence: sociable 
—unsociable, personal — impersonal, and warm — cold. It employed a semantic differential 
method with a bipolar 7-point scale. 


Note-taking Factor 


We created 6 items related to whether learners referred to the others’ writing in their 
note-taking. Moreover, we only asked the visual group 3 items regarding their feeling about 
using the visualization function of NoTAS. All items used a 5-point Likert scale (1 = strongly 
disagree, 5 = strongly agree). 


3.4. Guidelines for Analysis 


We compared the results of the two groups’ community awareness and social presence to 
investigate purpose 1, “Does NoTAS promote interaction with others?” We used these items as a 
measure to evaluate the promotion of Co-RL. 

We compared the note-taking factor between the two groups to investigate purpose 2, 
“Does NoTAS promote note-taking based on the others’ note-taking?” Moreover, we focused on 
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the visualization group results and compared them with the median of 3.00. The note-taking 
factor was used as an index to evaluate the nudge. 


4. Results 


In total, 33 participants answered the three questionnaires. The result of the 
questionnaires to evaluate Community Awareness, Social Presence”, and Note-taking Factors in 
the class are presented below. 


4.1. Community Awareness 


Table 1 shows the results of Mann-Whitney U test for the scores of community 
awareness. The index was reliable (Cronbach’s + = .88). The class in which learners used the 
visualization function of NoTAS is called “Visual,” while the class in which learners did not use 
it is called “Non-visual.” Twelve items found significant difference and marginally significant, 
visual was higher in all items. However, there were ceiling effects and floor effects except for No. 
7. Therefore, the learners felt that they were receiving real-time feedback on their note-taking by 
using NoTAS. 


Table 1. Comparison of community awareness 


Visual Non-visual (/- 
M, SDi M2 SD2 Mo 


1. I felt that learners in this class cared 260 135 1.89 128 0.71 90.50 * 
about each other. 


2. I felt that I was encouraged to ask 


2.53 1.19 2.22 1.00 0.31 114.00 0.16 


questions. 
a: Pte uneasy exposing Says MY 9g fg S00 1A OA T1200, «= OF 
understanding. 
4. I felt connected to others in this 400 1.13 111 032 2.89 200 ** 0.99 
class. (R) 
5. cs not feel a spirit of community. 353 136 1.61 0.78 1.92 34.00 “* 0.75 
6. I felt that this class resulted in only 
modest learning because of using 3.73 1.34 3.11 1.37 0.62 99.50 0.26 
NoTAS. (R) 
7. I felt that I received timely feedback 
on my notes and highlights in this 3.53 1.13 2.00 0.84 1.53 41.50 “ 0.69 
class. 
8. I trusted others in this class. 2.93 1.49 2.67 1.14 0.26 122.50 0.09 
9. I felt isolated in this class. (R) 4.27 1.10 2.94 1.16 1.33 56.00 ™ 0.59 
10. I felt that I could rely on others in 333 1.29 144 0.71 1.89 3400 “* 0.75 
this class. 
11. I felt that other learners did not help 413 0.92 2.72 123 141 49.50 “ 0.63 
me learn in this class. (R) 
12. I felt that members of this class 160 0.74 122 0.55 038 95.00 + 0.30 


depended on me. 
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13. Icould feel how the other learners 


were listening to the teacher's 4.20 1.27 2.50 1.30 1.70 46.00 0.66 
explanation in this class. 
14. I felt uncertain about others in this 353 125 2.78 140 0.75 94.00 0.30 


class. (R) 


15. I found that the other learners were 453 0.64 3.67 103 086 68.50 * 0.49 
taking notes very hard. 


16. I felt confident that others would nee 


3.00 1.20 1.61 0.85 1.39 49.50 0.63 
support me. 
17. I felt that I had enough opportunity 
to learn how to take notes in this 3.27 1.39 3.06 1.16 0.21 118.00 0.13 
class. 


18. Iwas curious about others’ note- 


taking behavior. 4.00 1.46 2.83 1.43 1.17 61.50 0.54 


HK 


Visual: n = 15, Non-visual: n = 18 Tp <.100, “p< .050, “p< .010, “p< .001 


(R): Reverse score, 5-point Likert scale 
4.2. Social Presence 
The social presence score was the average value of the three social presence items. The 
result of Shapiro-Wilk test showed that this score was normally distributed. Moreover, the result 
of Levene's test showed that this score was equality of variance. Table 2 shows the results of 


Student’s t-test for the social presence score. The index was reliable (Cronbach’s += .88). 


Table 2. Comparison of social presence 


Visual Non-Visal (/)- 


t 
M: SD: M2 SD2 M2 ° 
Social presence score 4.89 1.12 3.46 0.79 1.43 BQRAe 0.81 
Visual: n = 15, Non-visual: 1 = 18, Semantic differential method “p< .001 


As a result, the visual’s score was significantly higher than the non-visual’ score. 
Therefore, learners can feel a higher social presence of others by using visualization of NoTAS. 


4.3. Note-taking Factor 


We compared the six note-taking factors (No. 1 to 6) between the visual and non-visual 
groups. All items were significantly higher in the visual group. However, we found a floor effect 
for all items in the non-visual group for the note-taking factor. We expected this result because 
the learners who did not use the visualization function of NoTAS could not watch the others’ 
note-taking. 

Then, we compared the values for the visual group with a median of 3.00. Table 3 shows 
the results of One-Sample Signed Rank Test for the value of note-taking factors. The index was 
reliable (Cronbach’s + = .83). We asked only the visual group about No. 7 to 9. 
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Table 3. Note-taking factors 


M SD M-3.00 W y 
1. I wrote the "teacher's writing" based on the 
notes and highlights of other learners. BT hem er TROY a8 
2. I wrote the " teacher's oral explanation" 
based on the notes and highlights of other 2.80 1.47 -0.20 39.50 0.34 
learners. 
3. I highlighted "the important points" based on 333 1.45 0.33 58.00 0.03 


the notes and highlights of other learners. 
4. [highlighted "the points I did not 
understand" based on the notes and 2.40 1.06 -0.60 6.00 * 0.90 
highlights of other learners. 
5. I wrote "symbols" based on the notes and 


highlights of other leamers. pets Te READE “eS (Ot 
6. I wrote in "diagrams and tables" based on the 

notes and highlights of other learners. 200 ee vee 2000 Sa 
7. The visualization prevented me from writing 113 1.13 0.87. 1700 “ 0.72 

my notes. (R) 
8. The visualization prevented me from writing 

my hiphlights: (R) 2.87 1.60 -0.13 57.00 0.05 
9. enjoyed note-taking with the visualization 

function of NoTAS. 3.07 1.62 0.07 50.50 0.16 
n= 15, (R): Reverse score, 5-point Likert scale *p <.100, “p <.010 


As aresult, learners tended not to refer to others' highlights when they highligted parts 
they did not understand, W = 6.00, p < .100, 7 = 0.90. Furthermore, the visualization of NoTAS 
interfered with the learners' note-taking, W = 17.00, p < .010, r = 0.72. 


5. Discussion 
5.1. NoTAS Promotes Interaction Among Others 


The results of community awareness suggest that the learners felt the interaction with 
other learners by using the visualization function of NoTAS. For example, they felt connected to 
others and felt that other learners helped them learn and note-taking. Furthermore, the learners 
felt that they were receiving sequential feedback on their note-taking using the visualization 
function. On the other hand, learners did not feel that they had enough opportunities to learn 
note-taking even though they used the visualization function of NoTAS. We suppose that this is 
because this study was a short-term experiment. This result suggests that learners learn more 
about note-taking by using the visualization function of NoTAS in the long term. 

The social presence score suggests that the visualization function of NoTAS makes 
learners feel more social presence. Moreover, the visualization function helps learners recognize 
others’ presence and promotes interaction among learners. 

Therefore, we found that NoTAS promote Co-RL. 


103 


5.2. NoTAS Promotes Note-taking Based on the Others' Note-taking 


From the results of the note-taking factor, we found that learners who used the 
visualization function wrote their notes based on others’ writing more than the others who did 
not use this function. The visualization function of NoTAS automatically shares with learners the 
color and location information of others’ notes and highlights. These results suggest that learners 
write notes and highlights using this visual information. 

However, the mean scores of the visualization group were below 3.00 for many items. 
These results suggest that the visualization feature of NoTAS did not contribute much to the 
learners' note-taking. Furthermore, we found that the visualization might interfere with note- 
taking. There are two possible reasons as follow: 

@ The learners are not familiar with NoTAS 

@ The density of the visualization layer is set too high 

As a future task, we need to have learners use NoTAS for a long time and evaluate their 
note-taking. Furthermore, the density of the visualization needs to be reduced. 
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ABSTRACT 
This study investigates how collaborative concept mapping tasks affect the knowledge 
convergence of learners. Participants are divided into two groups by different group 
contingency, independent and interdependent, and we analyzed their discourses. We observed 
that agency could strongly affect their collaboration, but verbatim recitation was not a strong 
indicator of knowledge convergence in this context where learners had created an individual map 
before collaboration, and they used it as a reference. Though we also found the process of 
knowledge convergence, that did not strongly indicate that learners build knowledge together in 
both groups. Their actual collaborative map with an informational text was not considered as the 
knowledge convergence outcome. 


INTRODUCTION 

Online discussion, either asynchronous or synchronous, has been emphasized since social 
interaction was one of the critical capabilities for this new normal era, according to OECD 
(2019). However, efforts devoted to the study of group achievement and understanding have 
been significantly lacking, considering the new understandings built through dynamic interaction 
(Fisher & Mandl, 2005). This paper extends the investigation of in-depth learning, focusing on 
collaborative achievement, particularly knowledge convergence, while constructing a 
collaborative concept map. In this study, we engaged Information Science and Technology 
undergraduates in collaborative concept mapping to do the summary activity of a chapter, and 
we analyzed learners' knowledge convergence by different group contingency from their 
discourse. The finding from this study can inform the development of practical instructional 
support within the context of learning linearly arranged information. 
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THEORETICAL RATIONALE 

Dillenbourg (1999) defined collaborative learning as two or more people's attempts to 
learn something together compared to cooperation. Team members individually solve sub-tasks 
after splitting the work and assemble the individual's results for the outcome in cooperation, 
whereas individuals share and negotiate for the outcome in collaboration. Roschelle & Teasley 
(1995) also defined collaboration as a coordinated and synchronous group activity as a result of 
individual learners’ consistent effort to construct and develop a shared conception of a problem. 
Concept mapping is a visualizing technique to organize and represent the relationships among 
nodes(concepts) by edges (connecting nodes) representing relationships among the concepts as a 
network of ideas as a part of qualitative methods (Novak & Gowin 1984; Novak & Canas, 2008). 
Concept mapping tasks are generally regarded as cognitively demanding tasks given complex 
procedures such as identifying the main concepts and finding relationships among nodes by 
focusing on the organizational structure of the text, simultaneously screening the learning 
materials (Jonassen 1997; Hay, Kinchin, & Lygo- Baker 2008). These tasks generally improve 
verbatim knowledge and comprehension, and inferential skills for the contents (Novak & Gowin 
1984). Thus, as a learning strategy for knowledge construction, collaborative concept mapping 
can certainly enhance conceptual understanding (Stoyanova & Kommers 2002; Farrokhnia et al., 
2019). Collaborative concept mapping tasks are essential strategies to integrate individual 
learning with various group learning skills, such as creating a shared meaning of the task, 
concepts, procedures, and strategies for knowledge construction (Van Boxtel et al., 2002). 
Bereiter and Scaldamalia (2003), who appear as the pioneers in the field of knowledge 
construction, insist the creation of knowledge only occurs in collective processes, and Jeong and 
Chi (2007) indicated that in-depth learning only occurs after the group members integrate 
reasoning into personal understanding with the shared meaning. Knowledge convergence, 
essential to evaluate the depth of understanding of learners, mainly focuses on mutual influence 
through social interaction; as such, it is a process where two or more people exchange and 
converge their knowledge of the problem. Through mutual understanding, the knowledge finally 
becomes similar, which can be called 'shared understanding' of the content (Hutchins, 1991; 
Rogoff, 1998; Jeong & Chi, 2007; Roschelle 1992). 

Processes and outcomes are often used to explain knowledge convergence (Roschelle 
1992; Jeong&Chi 2007; Fischer&Mandl 2005). The process of convergence is conceptualized in 
various ways. One approach is based on knowledge contribution, which emphasizes that learners 
should contribute the ideas to varying or similar extents by counting the number of turns in 
discussion (Cohen 1994). This process can include grounding (Clark & Brennan, 1991), where 
both parties believe both understood and contributed to the discussion. The grounding process is 
the bottom layer of negotiation (Dillenbourg et al., 1996) and an essential part of achieving 
convergence. Shared knowledge, group mind, community memory, and team mental model are 
often viewed in the process of knowledge convergence as contributions of mutual knowledge. 
However, grounding may only capture the local convergence, not lead to a global convergence in 
terms of both mental models (Chi et al., 2004). Roschelle (1992) indicated that conversational 
analysis (CA) and pragmatics are critical in knowledge convergence research as interaction 
provides a means to construct abstract concepts collaboratively through the gradual refinement of 
ambiguous meaning. In addition to knowledge convergence processes, the other aspect to 
explain this phenomenon is resulting outcomes or mutual understanding. The outcomes can be 
defined in one way as "increased similarity in the cognitive representations of the group 
members" (Jeong & Chi, 2007, p 288) as their knowledge will be incrementally elaborated. That 
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is, learners mutually influence the knowledge outcomes of the group members; common 
knowledge or common ground are examples of outcomes of this mutual influence (Roschelle 
1992; Teasley 1997; Jeong & Chi 2007; Mercier 2017). In Roschelle (1992)'s study, the 
outcomes of two learners were gradually similar, indicating that the similar representation as an 
outcome after collaboration is convergence. For example, each learner interprets a situation, 
adjusts their understandings, and collaborates to solve problems; this process leads to the 
outcome of convergence. Jeong and Chi (2007) assessed knowledge convergence quantitively by 
conducting pre and post-knowledge tests to see the increase in common knowledge within the 
group; they defined knowledge convergence as an increase in common knowledge within a 
group. The results of pre and post-test performance revealed that learners shared more 
knowledge pieces and mental models after collaboration, though the association between the 
amount of interaction and the increase in common knowledge was not statistically significant. 
The study design considered the influences of learning artifacts (e.g., concept map) besides 
collaborative dialogues. Another study using pre and post test measures to quantify the 
knowledge convergence outcome differences when students had different goal assignments 
(either a learning goal or a performance goal); was conducted (Mercier, 2017). Though the 
results did not represent the differences in having learning goals or performance outcomes, there 
was a difference in knowledge convergence; groups with learning goals showed more knowledge 
convergence than groups with performance goals, suggesting that creating achievement goals for 
collaboration can influence interaction behaviors. 

Peterson & Roseth (2016) developed four CSCL (Computer Supported Cooperative 
Learning) strategies to increase students' cooperative perceptions based on the social 
interdependence theory: social interdependence, summarizing, scripts, and synchronicity. We 
particularly apply three relevant strategies to examine this small-scale research. Social 
interdependence (Johnson & Johnson, 1989, 2005) describes how students perceive their success 
as being affected by others' works, so hypothetically, the interdependent group has a more 
positive perception of their collaboration than the independent group. Additionally, requiring 
students to work on a collaborative summary should enhance collaborative perceptions because 
the shared goal involves active collaboration to create a group product (Ortiz, Johnson, & 
Johnson, 1996). Summarizing enhances achievement encouraging students to focus on the most 
relevant material and integrate it with existing knowledge (Hidi & Anderson,1986; Wittrock & 
Alesandrini, 1990), and using concept maps for summarization can even double the effects on 
text comprehension (Chang, Sung & Chen, 2002). In short, the greater positive goal 
interdependence by adding summarizing tasks with concept maps could enhance the benefit of 
collaboration, increasing the group productivity. Another essential part of this research is 
synchronicity. A synchronous video conferencing tool can enhance their collaboration by 
allowing them to convey social cues negotiating to construct knowledge in real-time, and 
researchers to analyze the processes of constructing a concept map during collaborative works. 
Therefore, it is critical to study knowledge convergence using a concept map with the concept of 
social interdependence in synchronous discussion in computer-supported learning to understand 
their deeper learning. This research investigates whether participants interact to share their 
individual knowledge and improve their conceptual understanding. Additionally, we explore 
their term usages to understand which terms are more frequently used together and how the 
terms in the interdependent group differ from the independent group. 
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METHOD 
It occurred within the Information Sciences and Technology introductory course of 80 
students at a Northeastern American university. For this particular paper, we randomly selected 
one independent and interdependent group to compare and analyze their discourses to understand 
how the collaborative concept maps are constructed differently by the group contingency to find 
evidence of deeper learning. 


Procedure 

Each group comprised three students, and they were randomly assigned to either an 
independent or interdependent group. The independent group students read one chapter about 
"system design and development" and constructed an individual concept map using the given 
tool, whereas the interdependent group students read one-third of the chapter and constructed an 
individual concept map using the same tool. After this preassigned task, both groups 
synchronously met and collaborated to create one map for the group. During the collaboration, 
students were required to create at least 20 nodes with edges but were not asked to consider the 
directionality of the nodes. Students were told their options not to be included in the research 
without any negative influences at the end. There was no time limit, and participants could look 
at other team members' maps. A tutorial of a tool, Cmap (Novak & Canas, 2008), was provided 
to instruct how to place words(nodes) with links (edges), including the directionality in advance. 


RESULTS 

Lexical network analysis 

Based on the content-related term frequency in two transcripts, we created their lexical 
networks in different group contingencies and compared them to understand the difference by 
the group. We picked 40 relevant terms from the most frequently used terms, and networks 
joined in pairs by edges were generated. Modularity, based on the eigenvectors of a 
characteristic matrix for the network, is a highly effective measurement technique in network 
analysis to detect delineated clusters (Newman 2006). With this definition, we found four 
different term clusters with 17 nodes and 51 edges in group 1(independent) and seven clusters 
with 22 nodes and 114 edges in group 2(interdependent). Simply describing, there are more 
nodes so as more clusters and the terms in each cluster were different in groups. Though each 
cluster includes identical terms, slightly different terms in clusters were found by the groups— 
also, the average degree and average weighted degree of the interdependent group was 5.182 and 
9.090, which is relatively higher than the independent group of 3 and 6.647. The graph density 
of both groups is low and cannot be compared because of the different numbers of nodes(Table 


1). 


Independent Interdependent 
Average degree 3 5.182 
Avg.Weighted Degree _| 6.647 9.091 
Network Diameter 5 4 
Graph Density 0.188 0.247 
Modularity 0.303 0.204 


Table 1 Lexical difference in groups 
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Discourse Analysis 

Roschelle (1992) indicated that conversational analysis (CA) and pragmatics are critical 
in knowledge convergence research as interaction provides a means to construct scientific 
concepts collaboratively through the gradual refinement of ambiguous meaning. In his case 
study, the dialogues of a pair of high school students were analyzed when they collaborate to 
learn the concepts in physics within the concepts of conversational action, conceptual change, 
and shared knowledge. 

Craig, Rick, and Julien (Group 1) and Malissa, Josh, and Bryce (Group 2) are 
undergraduates taking an introductory course in information science. The main difference in the 
tasks between the two groups is the content they read. The members of group | were assigned to 
read a whole chapter, while the members of group 2 were required to read one-third of the 
chapter. After that, group 1 individually constructed a whole map of a chapter, while group 2 
members created only one-third of the map before their collaborative work. The evidence of 
social outcome is shared knowledge as a process for knowledge convergence. During the 
collaboration, ideally, learners interpret a situation, coordinate their understandings, and come up 
with a solution to a problem together, and this process leads to the outcome of convergence. 
Here we focus on knowledge convergence in socially shared meaning, and we found knowledge 
convergence in both groups but could not conclude their collaborative map is their knowledge 
convergence outcome. 


Agency by grouping 

"Agency refers to the capacity of an individual or group to affect change on some entity, 
person, experience, state... and it is also connected with the notions of power and control 
(Strauss and Feiz, 2014, p.293)." Two groups were required to construct a collaborative map 
after different task requirements: group | members were asked to read a whole chapter, while 
group 2 members were to allocate their reading by one-third of the chapter per individual for the 
knowledge dependency. We expected group 2 to be socially dependent on their achievement as 
the outcome is more likely to be affected by other members' commitment and knowledge. 

Group 1, whose task was to read a whole chapter individually, caused to rise of a 
dominant leader. Grammatical and conceptual connotations of his utterance with controlling 
attitudes signified the role in the group. In their previous conversation, Craig asked what other 
nodes connected with the node of programming languages, and Rick suggested having 
programming as a node and branch off to higher-level language and lower-level language. In the 
excerpt below, Craig uses imperatives to provide direction, and the rest of the members agree 
and follow. When Rick suggests branching macro language off languages, Craig says, "that's 
what I will do." This sentence represents his way of agreement. Also, asking an obvious question 
is often used as a means of control. 


Group 1- Independent 

Craig — OK so then we’re over here. I’ll connect those in a second, just give me a sec. We’ ll 
put high level languages over here, and then while I’m connecting that you guys, uh, work on 
high level languages. We don’t need a ton, I mean as long as we get some. 

Julian — Yeah, um... low level...assembly...machine...low level...high level... [reading out 
of book to himself] 

Craig — OK, uh...high level languages...[mumbling a little] algorithm. Oh OK, Ricky. I got 
you. I see what you’re doing. OK, so high level languages... is that like basic and C and C++? 
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Rick — Yeah. 

Julian — Yes. 

Rick — You could also branch, uh, macro languages off of languages. 

Craig - OK, that’s what Ill do. Now do we have to cover the whole chapter? 


Group 1- Independent 

Julian - Did it — did it go across? It doesn’t matter about that. 

Craig — OK. Um, if you wanna move — Julius, if you wanna move Java up to the high level 
languages up there and I already got C++, so if you wanna move FORTRAN and- and Java up 
there, go ahead. OK Ricky, for macro you just had 4 generation and 5" generation? 

This excerpt above is a dialogue between Julian and Craig. While Julian was 
constructing a part of the map individually, his link(edge) crossed the part Craig was working on. 
He overly reacted and indirectly instructed Julian what to do, limiting his work, and promptly 
changed the attention by talking to Rick obstructing utterance of Julian. These two excerpts 
describe his dominant behaviors showing the agency of the group. Unlike our assumption that 
the independent group will be more collaborative than the interdependent group because of the 
different content distribution, the independent group worked more cooperatively, and the agency 
was affected. 

Group 2 represents distributed agency. Bryce interpreted the situation by asking a 
question about a task, and Josh provided solutions by mentioning other team members' maps 
shared in advance. In their decision-making process, they coordinated their understanding and 
came up with a solution how they wanted to work on, though the process took a long time. This 
is rather a collaborative process of decision-making than knowledge building of the learning 
content. Each member had authority or expertise in terms of the book knowledge because they 
all read the different sections of a textbook and constructed relevant parts of the map 
individually. 

Group 2- Interdependent 

Josh —so, if we just take systems design and development, put like a couple nodes off of it, 
then we can each tackle one of those nodes and just get the [Brian — Yeah.] high- the first two 
levels done — and then split off of a second tier. 

Melissa —Right. 

Bryce — Yeah. 

Josh — Is that the three main categ — is that what they’re — what they’ re - it’s how people make 
programs, [garbled] programming languages and methodologies, and what else is there? Or is 
that it? Programs and perspectives? Systems analyst — analysis — and systems [garbled]? 
Melissa — Yeah, it’s right here...three nodes. 

Josh —Then there’s the science of computing. The next one... 

Melissa — State of software. 

Bryce - Oh, I see what you’re doing. You’re going through each thing. 

Josh - Yup. And...that’s it. 

Bryce — That’s it. 

Josh — That’s it. 
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Bryce — Um...OK. So, Melanie, for yours system development [Melanie — Yeah.] and 
lifecycle, your main points are analysis — I’m gonna write these down — analysis, design, 
development... 

Melissa- Um, OK, sorry my screen is really small so I have to go back and forth. Um, analysis, 
design, development, implementation... 

Bryce- I'll help you. 

Josh - You want me to — [Melanie — Yeah.] is it cool if I start going from the beginning? III 
just start, start from the ‘how people make programs’? 

Bryce — Yeah, yeah. Um, retirement... um, implementation...oh maintenance, Melanie? OK 
and then... 

Melissa —Is that all we really need for that, or do we wanna go in-depth at all? 

Bryce — So with retirement we can do ‘in with the old’, the ‘out with the new’. 

Melissa — Let’s see... 

Bryce — Um, implementation...one of them is approaches to implementation. 

Melissa — Yeah. 


In the first excerpt, about 40% of the dialogue is to express agreement. Josh suggested 
creating a node, "system design and development" to begin, and the collaborative map was 
developed with each of the personal maps. After that, Bryce checked the assigned parts, and 
Josh created a few key nodes from his part, asking other key nodes to connect with. This excerpt 
reveals that Josh is leading the direction but not in a completely authoritative manner. The 
second excerpt represents Bryce helping Melissa to find relevant terms to connect with 
retirement and implementation. Again, there was no dominant leader in group 2. When the 
groups are compared regarding agency, the nature of group 2, distributed content expertise, made 
them cooperate more with positive social dependence. In contrast, the members in group | had 
the same content knowledge, even having a personal map of the content, so ideally, they were 
supposed to collaborate to negotiate the connection among nodes. However, instead, one 
dominant figure led the group conversation with relatively low social dependence. 


Verbatim knowledge as a strong indicator of shared knowledge 

Verbatim knowledge often refers to the items stated directly from the original text, such 
as remembering facts and stating knowledge as a lower order thinking, while inferential 
knowledge(conceptual knowledge) refers to the items needed to connect multiple concepts such 
as concept formation and problem-solving as higher-order learning (Clariana & Koul 2006; 
Blunt & Karpicke 2014). The concept mapping tasks require both verbatim and inferential 
knowledge given the complex procedures of identifying the main concepts and finding 
relationships among nodes by focusing on the organizational structure of the text while viewing 
the learning materials (Novak & Gowin 1984). 

The unique and confounding aspect of this dataset is that all participants have their 
reference maps when collaborating to construct the group map, but retrieving and reciting those 
individual verbatim knowledge maps cannot be the evidence of shared knowledge. Thus, the 
verbatim knowledge we consider here is only when they refer to the textbook. Both groups 
referenced their textbook but were rare because of the personal reference maps. The behavior 
referencing their textbook appeared when checking the connections among nodes. 


Group 2- Interdependent 
Bryce — What stuff? That’s um...programming languages and methodologies. 
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Josh — What does that connect to? 

Bryce — Uh...let me open my [cut out] page book. 

Josh — That go... 

Bryce —...connect off the center. I hate this thing. So people make programs, and then 
programming languages and methodologies. 


Group 1- Independent 

Craig- Is computer science and the science of computing — are they the same thing? 
Julian — Com... 

Rick — Wait, what’s that? 

Craig — Uh, if you — the science of computing and computer science. Are they one and the 
same? Are we saying the same thing there? 

Julian — Uh, the, um, that starts off the, um... it starts it off. If you look on 536 — 

Craig — Yeah. 

Julian -that’s the title: The Science of Computing. 


Knowledge Convergence 

Interesting part in this discourse data originates from the artifacts: individual concept 
mapping. As all participants construct their personal concept maps based on their assigned 
reading before joining a collaborative work, their knowledge should be reflected on those 
individual maps, which are frequently referred to during discussion in both groups. Thus, we 
could induce that their references in these dialogues are mostly from their personal maps and 
could assume that when participants refer to their own maps during the group map construction 
(identifying the concept), negotiation occurs, and their group maps reflect their negotiation 
(finding relationship among nodes), we could see them as a process of knowledge convergence. 


Group 2 Interdependent 

Josh - ...going to connect this here. This guy really needs to go here. 

Bryce — What’s up? 

Josh - I, I’m only, I, I’m double linking up here...going crazy. 

Bryce — Wait why are you double linking? 

Josh — I — cause it make sense, I guess? I don’t know where I’m going...[garbled] cross the 
stream. 

Bryce — Do that magic thing to it. 


In the excerpt above, Josh tried to double link with one node, and Bryce asked why he wanted to 
do it, which could be evidence of negotiation if Josh explained the reasons, but Josh intuitively 
did it saying "cause it make sense, I guess". To them, the auto-format function, "magic thing," 
seems like a convenient backup skill to verify their map. This excerpt is interesting considering 
the aspect of knowledge convergence: Josh brought a double linking problem and cannot explain 
the reason. Josh and Bryce both could have referenced the textbook, but instead, Bryce suggests 
auto-formatting, which might solve the problem without the retrieval process, whether that is 
either verbatim or conceptual knowledge. Thus, we cannot find the clues of knowledge 
convergence. 


Group 2 Interdependent 
Melissa — What still needs done yet? 
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Josh -...program with machine language. 

Bryce — Are we going to put everything underneath, or...what did you guys wanna do? I guess 
we'll... 

Bryce —I guess we can let it auto format? Or do you wanna — 

Melissa — Well the systems lifecycle should be closer to the bottom cause that was like the end 
of the chapter. 

Bryce — What? Oh are you saying this — is this wrong? Well would — I thought this would be 
the center and then there would be arrow pointing to each. Is that incorrect? 

Melissa — No, no, I mean that’s fine. It’s just like the systems development comes after the 
programming languages — in the chapter, at least. 

Bryce —I think we’re good. 

Melissa — Yeah. 

In the excerpt above, they are checking the parts in the map not completed yet. Josh 
found a part (program with machine language) that needs attention, and Bryce suggested 
intuitive solutions, either putting everything underneath based on his knowledge or 
autoformatting. Melissa doubted his solution as she thought the node should go down to the 
bottom of the map because she assumed the terms in the textbook are chronologically delivered, 
which is incorrect. Bryce expressed his thought with reasoning, and Melissa did not rebut the 
point Bryce made but confirmed her understanding was slightly inaccurate in that system 
development should come after the programming languages. This dialog looks like a weak 
version of the negotiation process. 


Group 2 Interdependent 

Josh — It’s gotta go there like that. That guy’s gotta go there. [last part garbled] ...get this over 
here...and then put this one... 

Bryce — Do you just wanna do the [middle part cut out] the whole thing? 

Josh — Uh it gets, I mean if we do it, we can try- we can try it out— Whoa! Whoa, what’s go- 
what just happened? Hold on. 

Melissa — Oh no. 

Josh — That made no sense. Yeah that got bad real quick. Um the auto-format thing can 
sometimes mess up, so if we do it we gotta just one person do it so they can -can- they can 
undo it if it doesn’t look right. But I think we’re looking kinda good. It’s pretty good. Well 
except for that one down there. 

Before this excerpt above, the auto-formatting function had helped them organize the 
map, which led them not to deal with complex conceptual understanding connecting distributed 
concepts. Josh structured the map, and Bryce wanted to use the autoformat function again to 
make sure Josh was right. They realized this function does not always provide the solution 
indicating this team could at least notice when the map goes wrong. Each member's knowledge 
was not entirely engaged in this collaborative concept mapping process: they tried to solve 
problems working on the group map but did not show deeply engaging conversation using their 
individual knowledge. 


Group 1- Independent 
Craig — OK Rick, would you say macro languages off of languages as well? 
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Rick — Yeah...so then there’s um, you can have 4" generation languages and 5" generation 
languages. 

Craig —for macro you just had 4" generation and 5" generation? 

Rick — No, uh, make, make those just go off of languages too. 

Craig - Oh just — oh just keep macro languages by itself then? 

Rick — Yeah. 

Craig — OK. 

Rick — And under 4" generation you could have query language. And that’s all I had for — 
that’s all I had before for that. 


Craig - OK and you want me to do languages by itself? 

Rick — Yeah like have, uh, programming like branch off of languages. 

Craig — Like that? 

Rick — Yeah and then connect them. Yeah. 

Craig — OK, OK. So we do programming, we do languages. I remember there’s — there’s like 
what? There’s hi — there’s, uh, low level and high level languages, right? 

Rick — Yeah. 


In the above two excerpts, members structurize the map to organize terms. Their 
dialogues above are questioning from Craig and answering and confirming from Rick. Craig 
primarily relies on the opinions of Rick, and Rick refers to his personal map. There is no further 
negotiation process as proof of converging knowledge, so their symmetry of knowledge in the 
content seems slant toward Rick even though all are supposed to read a whole chapter. 


Group 1- Independent 

Craig- computer science as one and then, um, off of computer science I actually had a ton, so 
I'll work on that and then the other- the other big one that I had was system development 
lifecycle. I’m not sure if you guys had that or not. 

Rick — Wait, what was the first one you had? 

Craig — Computer science. 

Rick — OK. 

Craig — And then off of that was like I had — uh, actually from that one right there, that node — 
I had computer theory, algorithms, data structures, uh, programming concepts and languages, 
management information systems, software engineering, and computer architecture. There’s a 
ton. 

Rick — Yeah... Maybe we should put a node for like, um, chapter four — the name of chapter 
14 somewhere? Then we could connect like the two main ones to that. 

Julian — That’s — that’s what we started with. 


In comparison to the interdependent group (group 2), the collaboration of group 1 was 
superficial even though we assumed they were supposed to have an in-depth discussion as all 
members read the entire chapter. For example, some cognitive outcomes, a conceptual change 
based on the verbatim knowledge from the textbook, were expected by in-depth discussion to 
construct the map but not observed. There were some moments of shared knowledge, such as 
checking where each one is at in order to discuss from there, but that did not clearly represent 
knowledge convergence. The shared knowledge here is considered as a contribution of mutual 
knowledge in the process of grounding. 
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When it comes to the Cmap tool, group | did not desire to explore other functions besides 
connecting nodes to construct a map, whereas group 2 used the tool effectively to connect a 
chunk of nodes to another chunk though it became a time-saving tool not to do complex 
thinking. Whenever they were in problems connecting vertices, they agreed to use that auto 
format though they eventually learned this function did not always work. Finding relationships 
among nodes by focusing on the organizational structure of the chapter involves inferential and 
higher-order thinking. 


Agency in knowledge convergence 

We examined how the distributed agency affects the knowledge convergence compared 
to the concentrated agency. A context engenders different levels of collaboration: the 
collaborative work is more likely to be effective with people having a similar status and action 
(Dillenbourg, 1999). The agency particularly influenced the symmetry of action in this case. 
Craig's controlling and initiative manners led to unbalanced communication in collaboration. 
For example, turn-taking between Craig and Rick and Craig and Julian was more than 95%. 
From the beginning, Craig referred to Rick's map complementing and asked his opinions, and 
those Q&A style interactions were maintained until the end. His questions at the beginning were 
to distribute their mapping tasks and later to ensure specific nodes with links were right. Craig 
mainly initiated conversation without intent to learn; he has never tried to validate reasons or 
rationalize the concept, so there seem rare opportunities to converge their knowledge while 
constructing the map. This discussion, driven by one leader with this type of conversation, Q&A, 
did not lead members to converge their knowledge to identify the main concepts and find 
relationships among vertices. 

However, the members of the interdependent group with the relatively distributed agency 
were more actively participated in elaborating the map by finding relationships referencing their 
textbooks. This distributed agency positively affects knowledge convergence. There were 
conflict-oriented consensus and integrate-oriented consensus (Weinberger and Fischer, 2006). 
For example, when Bryce asked the overall structure of the map to connect different chunks of 
nodes, two members replied, and that process went on for a couple of minutes (shared 
knowledge) while looking at the common map to decide the path of some nodes. Even though 
certain words were not explicitly spoken, interlocutors understood, and they maintained their 
interaction. Regarding the tool, they had a good collaborative process while exploring the tool; 
Josh tried to use a specific tool function, auto-format, to organize it. Melissa knew how to do it, 
and Bryce was interested in applying it. 

Group 1- Independent 

Craig- Um, tell me what goes, uh, Julius — what I would go to from low level languages? 
Rick — Well low level’s, um, machine and assembly languages. 

Julian — Yeah I see machine, yeah. 

Craig — OK hold on. 

Julian — It says, it says, uh, machine, assembly, assembler, and low level. 
Craig — OK. 

Julian — And high level is completely separate. 

Craig — OK. Machine is one — 

Julian — So you could — 

Craig — What — what are the other ones? Assembly? 
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Julian — Yes. 

Rick — Yeah. 

Craig — And what was the third one? 

Julian — Um... low level. 

Craig — Oh I thought machine and assembly were low level? Oh I think that’s it. 


Rick — Those are the only two that — those are the only two I had in mine. 


CONCLUSION 

Online discussion, either asynchronous or synchronous, has been emphasized as an 
effective instructional strategy. In this research, we tried to investigate knowledge convergence 
while constructing a collaborative concept map, and little knowledge convergence as a clue of 
deeper learning in these two cases was found. The required individual task, drawing a concept 
map in advance, influenced collaborative work as most students rely significantly on their 
individual maps regardless of their groups, which is likely to reduce the situation for socially 
sharing meaning. Having an additional artifact, a concept map, obstructs students’ discussion, so 
even if there was knowledge convergence, that was not observed. Also, creating a concept map 
with the unfamiliar tool was already an arduous task whose cognitive loads are probably 
negatively affected. That is, there were too many confounding factors to affect their knowledge 
convergence in their collaborative work. Instead of negotiating to connect nodes, students just 
used a certain function of the tool to avoid complex thinking, intuitively connected nodes, and 
passively accepted others' thoughts. Furthermore, knowledge convergence was not frequently 
observed in collaborative work when the content builds foundational knowledge. The group 
difference reveals a slight difference in their collaborating process in extent, but we could not 
find an absolute difference in knowledge convergence by analyzing the discourse in terms of 
grouping. 

Finally, a verbatim recitation of a concept or inference rule was searched as this will be 
the strong evidence of shared knowledge. Their verbatim recitations were either from the 
textbook or their individual maps, but these recitations were not evidence of shared knowledge. 
Those citations were only the references. We assume it is because the content itself is not 
meaningfully interrelated but linearly arranged to the novice, unlike the concepts of Physics 
(Roschelle 1992). There still can be a negotiable place for knowledge convergence, but 
participants should understand the contents more thoroughly to interactively communicate for 
negotiating to link the nodes to reach that point. Future research will be needed to focus more on 
the attributes of knowledge convergence and design the intervention accordingly. 
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Abstract 


Online communities of practice (CoPs) in Instructional Design play an important role in creating 
and transferring professional knowledge (tacit knowledge). This study examines the 
characteristics of professional knowledge from 4,717 anonymized users’ posts available in the 
Instructional Designer Facebook group from August 3rd, 2017 to September 9th, 2020. This 
group is the largest public group on Facebook with 17,640 members as of September 29th, 2021 
(About Us CrowdTangle, n.d.). Natural language processing approaches, including n-grams, 
sentiment analysis, visualization of named entities, and topic modeling, were implemented to 
extract tacit knowledge from written forms of communication. Three topic models, including 
bag-of-words, TF-IDF, and sentence embeddings, showed topics related to corporate training, 
online training, authoring e-learning tools, resource downloads, virtual reality, mobile learning, 
and outsourcing e-learning development. In the visualization of relationships between named 
entities, this particular CoP is characterized by (1) the recognition of members’ contribution and 
request for feedback, (2) the use of hashtags related to e-learning development and trends, and 
(3) the request for career advice related to the Instructional Design profession. The study has 
implications for developing practitioners’ pedagogical proficiencies and establishing community 
knowledge management and curation practices. 


Keywords: Communities of Practice, Natural Language Processing, Social Media 
Introduction 


Tacit knowledge, or tacit knowing, was initially coined by Polanyi (1983) to describe the 
type of knowledge that is difficult to transfer to another person. Lave and Wenger (1991) stated 
that CoPs are characterized by a shared domain of interest, joint community activities, and a 
shared domain of practice. While present studies examine Instructional Designers’ professional 
development needs and roles in academic and corporate settings using observation and survey 
methods, studies examining sources of articulated professional knowledge (explicit knowledge) 
in a virtual environment are required to understand current knowledge structures and gaps in 
Instructional Design communities. In the knowledge management literature, empirical studies in 
tacit knowledge extraction use n-grams and topic modeling algorithms to capture semantic 
relationships from explicit knowledge. The study uses natural language processing approaches, 
including topic modeling techniques to make topic representations and word sequences from 
users’ posts. 
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Literature Review 


The following literature review identifies the ways technology supports online 
communities of practice, the collaborative context using information and communication 
technologies (ICT), and the characteristics of Instructional Design communities. 


Technology to Support Communities of Practice 


With the advent of more collaborative ICT in Web 2.0, members in a CoP can co- 
produce, organize, discover, and share knowledge within a collaborative online context. Using 
ICT to support knowledge production and collaboration, CoPs have become increasingly virtual 
or Internet-mediated communities of practice. Rheingold (2000) initially coined the term virtual 
community to emphasize the evolution of ICT technologies and human-computer interactions 
that connect individuals with organizational structures. Online communities are defined as 
“online social networks in which people, with common interests, goals, or practices, interact to 
share information and knowledge, and engage in social interactions” (Chiu et al., 2006, p. 1873). 


Akoumianaki (2011) argued that technologies must support virtual communities of 
practice regarding functional and non-functional requirements. Functional requirements are the 
critical pieces of technology that support members’ participation and engagement. In contrast, 
non-functional requirements refer to the supporting technological mechanisms that enable 
participants to facilitate and refine their knowledge exchange. According to Heap and Kelly 
(2004), technical means must support the core function of knowledge creation and sharing of 
online communities regarding functionality, integration, usability, security, performance, 
reusability, and support. When these mechanisms are present in online communities, ICT tools 
support members in four ways; (1) ideation, information and document sharing, (2) contact 
management, (3) messaging and discussion, and (4) meeting and conferencing (Heap & Kelly, 
2004). 


As knowledge consumers, Nilan et al. (2004) stated that users in online communities try 
to make sense of information within a specific search context. The design of online communities 
can either facilitate or hinder users’ movement within the community. In their findings, Nilan et 
al. (2004) found that members spent most of their time searching and reading within the 
community. Examples of searching within the community included scanning for topics or news 
topics, checking out other users, selecting or clicking on options, and reading within the 
community (e.g., reading existing content, announcements, and responding to others). The 
authors also reported the hindrances that users faced in virtual communities. Poorly designed 
interfaces hindered users’ iterative searching, reading, and writing/talking within communities. 


Instructional Design and E-Learning CoPs and Competencies 


Schwier et al. (2004) argued that Instructional Design CoPs are born of convenience that 
allows informal engagement to solve specific project challenges or issues. The authors also 
investigated the features of Instructional Design CoPs in terms of history and culture, mutuality, 
plurality, knowledge repository, and tacit knowledge. They found that shared history and culture 
were not prominent features in Instructional Design CoPs. In contrast, passive participation as a 
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spectator was a critical element aligned with practitioners’ agendas and community values. In 
terms of mutuality, community members developed their protocols for contribution and 
interaction with others. At the same time, community participation was based on the plurality of 
the intermediate relationships with other members (i.e., experts in the field) that provided a wide 
range of considerations and solutions to learning problems. 


Due to the absence of a recognized accrediting body that identifies the required 
competencies for Instructional Design and Technology professionals, professional organizations 
have developed the competencies that define the knowledge, skills, and abilities of professionals 
in the field. Though several professional organizations developed their competencies, three 
prominent professional organizations use competencies to encapsulate the professional 
benchmarks, responsibilities, and capabilities of these professionals serving in different roles 
(e.g., Training Manager, Evaluator, Instructional Designer, and Instructional Technologist). 
These organizations are the American Talent Development (ATD), the International Board of 
Standards for Training, Performance, and Instruction (ibstpi), and the Association for 
Educational Communications and Technology (AECT). 


Problem Statement 


The characteristics of professional knowledge creation among Instructional Design CoPs 
in virtual environments are primarily unknown in the instructional design and technology 
literature. While present studies examine Instructional Designers’ professional development 
needs and roles in academic and corporate settings, exploring sources of professional knowledge 
in virtual environments is required to understand the current knowledge structures and gaps in 
Instructional Design professional knowledge. Professional knowledge (tacit knowledge) is 
conveyed and shared in explicit or written form for and by community members. 


Purpose & Significance of the Study 


The purpose of the study is to extract tacit knowledge at the externalization stage (i.e., 
articulated professional knowledge) from users’ written professional knowledge who are 
members of the Instructional Designer Facebook group. In the externalization stage articulated in 
the SECI model by Nonaka and Takeuchi (1995), explicit and tacit knowledge is generated, 
transferred, and recreated in organizations through socialization, externalization, combination, 
and internalization processes. The study explores the following research questions: 


e RQ1: What are the most frequent words and word sequences used in the CoP? 

@ RQ2: What are the characteristics of sentiment, named entities, and relationships among 
entities in the CoP? 

e RQ3: What latent topic structures exist in the CoP? 


The significance of this study involves a community’s ability to manage their existing 
body of knowledge and leverage members’ collaboration to generate new professional 
knowledge. CoPs act as knowledge stewarding communities where members can organize and 
manage a body of knowledge from which the community can draw professional learning to 
improve their practice. CoPs also act as a crowdsourcing mechanism where community members 
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can generate professional knowledge by converting tacit knowledge, or know-how experiences 
in the field, into explicit forms (e.g., written texts, videos). 


Theoretical Framework 


Research studies in the knowledge management literature examine tacit knowledge 
extraction from explicit forms of knowledge in the workplace (e.g., online platforms, documents, 
and e-mail communication). These studies are explored through the SECI model, where 
knowledge is continuously created through socialization, externalization, combination, and 
internalization phases (Nonaka & Takeuchi, 1995). First, tacit knowledge is created through a 
socialization process, and its tacitness is difficult to codify into explicit knowledge. Second, tacit 
knowledge is externalized or articulated in symbolic language for sharing with other groups or 
individuals. Third, the combination step requires applying and reorganizing explicit knowledge. 
Fourth, when explicit knowledge is applied, individuals embody the knowledge as tacit through 
action and reflection. 


Methods 


The Instructional Designer Facebook group was chosen as a data source for this study. 
This public Facebook group was founded in 2011 by the E-learning Industry website. According 
to CrowdTangle Intelligence, a public insights tool from Facebook, the Instructional Designer 
group is the largest public group on Facebook with 17,640 members as of September 29th, 2021 
(About Us CrowdTangle, n.d.; Instructional Designer, n.d.). Any Facebook user can request 
access to this group by answering a filter question related to their reason for joining the group. 


A total of 6,760 anonymized users’ posts from August 3rd, 2017 to September 9th, 2020 
were extracted with Python scripts. After cleaning Facebook posts that contained “hi,” “hi there," 
“hello all,” and “good morning professionals,” the dataset was reduced to 4,717 posts. Then a 
sentiment analysis was performed using the Textblob Python package. (TextBlob: Simplified Text 
Processing — TextBlob 0.16.0 Documentation, n.d.). Posts were also analyzed with a variety of 
natural language processing tasks, including (1) n-grams with NLTK, (2) Name Entity 
Recognition (NER) with the spaCy Python package, and (3) topic modeling using the Latent 
Dirichlet Allocation algorithm (LDA) and BERT (Bidirectional Encoder Representations from 
Transformers) (Linguistic Features - SpaCy Usage Documentation, n.d.; Natural Language 
Toolkit — NLTK 3.5 Documentation, n.d.; Open Sourcing BERT: State-of-the-Art Pre-Training 
for Natural Language Processing, n.d; NetworkX — NetworkX Documentation, 2014). N-grams 
are a sequence of words that predict the probability of the occurrence of words in the corpus. 
NER is a natural language processing task that locates named entities in texts into eight 
predefined categories or tags, including geographical entity, organization, product, person, time 
indicator, artifact, event, and natural phenomenon. 


The known limitation involved the removal of external resources placed in Facebook 
posts, including shared resource documents, recommended articles, and video links. While the 
tacit knowledge representations of these communities are static for a given period, topic models 
will continue to evolve as members engage in knowledge creation and sharing in their respective 
communities. 
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Findings 


After cleaning Facebook posts from 6,760 to 4,717, the study results are summarized 
below by research question. 


RQ 1: What are the most frequent words and word sequences used in the CoP? 


The average word count was 38.75 words, whereas the average sentence count was 3.14 
sentences in the Instructional Design Facebook Group. In Figure 1, the most frequent words 
included “anyone,” “id,” “course,” and “looking.” In terms of n-grams, the top three most 
frequently occurring bigrams were “instructional, designer,” “instructional, design,” “hi, 
everyone,” and “im, looking.” The top three most frequently occurring trigrams were “elearning, 
elearningtrends, elearningdevelopment,” “hi, everyone, im,” “instructional, design, technology,” 
“thanks, advance, hi,” and “would, love, hear.” As shown in Figure 2, the most frequent 4-grams 
were “follow, u, learningpark, learn,” “elearning, elearningtrends, elearningdevelopment, lm,” 
and “learnandgrow, learnfromhome, learner, learningeveryday.” 


Figure 1 
Instructional Designer Facebook Group Word Frequencies 
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Figure 2 


50 Most Frequently Occurring 4-Grams 
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50 Most Frequently Gocuring 4-grams 
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RQ 2: What are the characteristics of sentiment, named entities, and relationships among 
entities in the CoP? 


The sentiment analysis indicated that most posts (3,298) had a positive sentiment, 
whereas other posts were either neutral (1,011) or negative (408) sentiment. In terms of 
recognized named entities, most entities were related to the organization entity. In Figure 3, most 
organizational entities were related to LMS, elearning, Captivate, and Articulate. The limitation 
of NER involved the lack of differentiation between entity types related to organization, product, 
person, and country or nationality entities. For example, NER assigned person and nationality 
entities to web conferencing tools, authoring software, and learning management systems. In 
terms of product entities, shared resources (e.g., e-books) were also recognized in the 
community. The entity relationships with the highest degree included “educational, animation,” 
“instructional designer, e-learning industry,” “responsive e-learning design, multidevice,” and 
“Just for fun, effective learners.” 


Figure 3 


Most Frequently Recognized Entity Types 
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The most popular entities included “one” and “today,” whereas the most popular target 
entities had “good one” and “https.” The most popular relationships among entities were 
“thanks,” “thank” and “want.” When visualizing “thanks” as a popular relationship, as shown in 
Figure 4, members in this CoP appeared to engage with each other’s requests related to various 
topics, including resource sharing, Instructional Design graduate programs, and e-learning 
authoring software and hardware. 


Figure 4 


Most Popular Relationships for “Thanks” Entity 
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RQ 3: What latent topic structures exist in the CoP? 
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The bag-of-words, TF-IDF, and sentence embeddings topic models were developed to 
identify latent topics from the Facebook group. After obtaining the highest semantic coherence 
of 0.48 using five topics for the LDA algorithm topic parameter, as shown in Figure 5, the five 
topics in the bag-of-words and TF-IDF models had a similar distribution of topics related to: 


Instructional Design job postings 

Learner or student online training 

Asking for advice related to e-learning authoring tools 
Online course and video development 

Sharing of webinar events 


OY ee 


Figure 5 


TF-IDF Model Topic Distribution 
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In the last model using sentence embeddings with BERT, the model automatically 
performed a dimensionality reduction and clustering of texts. Based on the intertopic distance 
map, the 74 topics were present in nine clusters. In Figure 6, the nine topics with the highest 
probability were as follows: (1) Microsoft products, (2) lectora and storyline e-learning 
authoring, (3) virtual recruiting, (4) story articulate, (5) conferences, (6) Instructional Design, (7) 
managerial/trainer jobs, (8) instructional writing, and (9) selling online courses. 


Figure 6 


BERT Topic Probability Distribution 
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Discussion 


In the analysis of 4,717 users’ posts from the Instructional Designer Facebook group, the 
average sentence word count was 38.75 words, and the average sentence count was 3.14 
sentences. In terms of sentiment, 3,298 posts were positive, 1,011 posts were neutral, and 408 
posts were negative. Words with the most frequency included: “id,” “course,” “learning,” 
“anyone,” and “looking.” The most frequent 4-grams were “follow, u, learningpark, learn,” 
“elearning, elearningtrends, elearningdevelopment, Im,” and “learnandgrow, learnfromhome, 
learner, learningeveryday.” The entity relationships with the highest degree included 
” “responsive e-learning 
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“educational, animation,” “instructional designer, e-learning industry, 
design, multidevice,” and “just for fun, effective learners.” 


In this particular CoP, there was an emphasis on learning management systems that 
support various tasks, including task management, employee onboarding, and compliance 
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training. Based on the observed n-gram sequences and entities, the Facebook group posts 
appeared to suggest three aspects of knowledge sharing among members: 


1. Recognizing members’ contributions and requests for feedback through informal 
salutations and following certain users and posts 

2. Using hashtags related to e-learning development and trends for resharing and following 

3. Requesting career advice related to transitioning to the Instructional Design field 


Although topic representations in the bag-of-words and TF-IDF topic models were 
almost identical using the LDA algorithm, LDA relied on semantic similarities based on words 
rather than the context of the words. To compensate for the shortcomings of LDA, BERT 
allowed for discovering subtopics and topic outliers based on the context of the surrounding 
words. In the BERT topic model, nine topics were prominent in the Facebook group, including 
(1) Microsoft products, (2) lectora and storyline e-learning authoring, (3) virtual recruiting, (4) 
story articulate, (5) conferences, (6) Instructional Design, (7) managerial/trainer jobs, (8) 
instructional writing, and (9) selling online courses. 


Implications & Future Research 


The empirical evidence of tacit knowledge structures informs researchers and 
practitioners about the present capabilities and strengths of CoPs in producing and sharing 
knowledge in certain areas of Instructional Design expertise. Other research can build on new 
ways of creating and supporting new tacit knowledge production in virtual CoPs by 
experimenting with content curation practices, knowledge discovery mechanisms, repository 
tools, and social knowledge representation methods described by Cagliero and Fiori (2012) (e.g., 
subject-based classification, folksonomy, and structured knowledge). It is recommended that 
Instructional Design CoPs adopt a taxonomy or classification system based on professional 
competencies to organize existing professional knowledge and improve knowledge discovery 
within the community. Even though the primary function of this CoP is to support educational 
technology needs, building pedagogical proficiencies in Learning Sciences, Instructional Design, 
and Knowledge Management is also essential for practitioners’ professional development. 


Conclusion 


The resulting study contributes to understanding the knowledge production capabilities 
and shared practices in Instructional Design CoPs. After analyzing 4,717 anonymized posts from 
a public Facebook group in Instructional Design, popular word frequencies and n-grams were 
related to asking peers and looking for solutions. While named entities were related to learning 
management systems and e-learning tools, the network visualization of these entities showed 
active engagement in seeking advice related to resources, Instructional Design graduate 
programs, and software tools. 
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Abstract 


As educational institutions switched from in-person to Remote Learning or Emergency Remote 
Teaching (ERT), this study explores the search popularity indices of three search terms globally 
and in the USA from January 2020 - April 2021. Google search terms, such as COVID-19, 
Instructional Design, and Remote Learning, showed considerable search interest in Google 
Trends at the peak of the pandemic on Mach 11, 2020 (WHO, 2020). Pytrends was used to 
extract search interest of the primary and related terms from Google and YouTube Search 
globally and in the USA (Pytrends, n.d.). The search interest of the three search terms and 
associated terms exhibited similar peaks and dips of interest at the beginning of the pandemic 
and the 2020-2021 school year. Educational technology tools, including SeeSaw Learning and 
Zoom, were present as related search terms for Remote Learning. Search queries in YouTube 
revealed educational channels that host various lecture materials related to K-12 and professional 
development. The interactive visualizations in Tableau Public enable users to explore the search 
trend patterns of the three main terms and related queries. This project serves as an archive of 
users’ conscious information seeking to highlight the pivotal roles of Instructional Design and 
Remote Learning observed in search popularity indices in light of the pandemic. The project can 
be accessed at edtechtrends.javierleung.com or on Tableau Public (Leung, n.d-a, n.d.-b). 


Keywords: Instructional Design, Remote Learning, COVID-19 
Introduction 


The World Health Organization (WHO) declared the novel coronavirus (COVID-19) 
outbreak a global pandemic on March 11th, 2020 (WHO, 2020). Since the beginning of the 
pandemic, 1.3 billion learners are still affected by the school, and university closures as 
institutions implement entirely online and hybrid solutions (UNESCO, 2020). As instructors in 
higher education and K-12 adapt their face-to-face curricula to distance delivery formats, the role 
of Instructional Design and Technology has been a critical element in creating equitable access 
to learning experiences for all kinds of learners. 


This study reports on a data visualization project of global and USA search trends related 
to Instructional Design, Remote Learning, and COVID-19 before the pandemic and at the peak 
of the pandemic across Google and YouTube Search. This project explores popularity indices in 
a single interactive interface using Tableau Public by querying the Google Trends API. The 
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project also enables users to explore search popularity indices of multiple search terms over time 
from January 2020 through April 2021. In these visualizations, the audience can interact with the 
search popularity indices for trending searches and related queries for three search terms (i.e., 
Instructional Design, Remote Learning, and COVID-19). Although Google Trends provides a 
way for exploring the popularity index (0% - 100% being the most popular) of an individual 
search term for the past 12 months or since 2004 for specific search terms, the interface of 
Google Trends does not allow for exploration of multiple related terms. This project provides a 
better understanding of the pivotal role of Instructional Design, Remote Learning, educational 
technology tools in the context of the COVID-19 popularity index. 


Literature Review 


The following literature review is divided into two sections. The first section describes 
the needs of educators in ERT during the COVID-19 pandemic. The second section describes the 
applications of Google Trends in academic and clinical research. 


Emergency Remote Teaching and Learning 


During the Spring of 2020, K-12 schools and institutions of higher education moved 
face-to-face classes to online formats to prevent the spread of the virus that causes COVID-19. 
As the 2020-2021 school year has progressed, schools have faced the difficult transition to 
emergency remote teaching (ERT) while keeping students, faculty, and staff safe. Hodges et al. 
(2020) clearly defined the differences between ERT and online learning. ERT is the shift of 
instructional delivery to alternate delivery modes due to crisis circumstances. In contrast, online 
learning involves design decisions for effective online learning based on Instructional Design 
choices that generally require six to nine months before the course is delivered (Hodges et al., 
2020). Furthermore, Barbour et al. (2020) identified the four phases of educational institutions 
responding to the COVID-19 pandemic. The first phase involved a rapid transition to remote 
teaching and learning where the health and safety of students and educators were the most 
critical aspect while transitioning course materials to synchronous (e.g., video over Zoom or 
WebEx) and asynchronous (e.g., Google Slides and VoiceThread) online formats in four weeks. 
The second phase required ERT to include essential components, including (1) course 
navigation, (2) equitable access, (3) student support mechanisms, and (4) academic integrity. As 
face-to-face courses were transitioned to ERT with acceptable mechanisms for delivering quality 
learning experiences, the third phase involved careful planning in supporting students and 
educators for a full term for online delivery. In the fourth phase, K-12 schools and institutions of 
higher education had new levels of online learning infrastructure to support students and 
educators. 


Supporting the Needs of Educators 


Recent studies have identified the needs and challenges of educators (e.g., faculty in 
higher education) and staff (e.g., Instructional Designers and Technologists) in delivering ERT to 
students and supporting the professional development needs of educators in the light of the 
ongoing global COVID-19 pandemic. Redstone and Luo (2021) reported the types of 
professional development (e.g., teaching, community, and organization support) that instructors 
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in higher education needed during ERT. Their principal findings included a centralized 
information hub for resources to support ERT transition and community support for self-directed 
and peer-to-peer social activities (Redstone & Luo, 2021). Also, Vollbrecht et al. (2021) 
identified design considerations for ERT delivery. Seven suggestions include (1) considering 
learners in different time zones, (2) team-based teaching in synchronous sessions, (3) increasing 
faculty availability for student questions, (4) providing specific time for troubleshooting 
technical difficulties and classroom breaks in synchronous sessions, (5) providing several 
learning opportunities in both synchronous and asynchronous formats, (6) considering the well- 
being of students and educators, and (7) findings ways to engage students virtually. 


Google Trends for Research 


Google Trends is widely used to complement traditional academic and clinical research 
to predict or forecast outcomes (e.g., economic activity, disease tracking, and user’s search 
behavior) in a given geographical area (Levinthal, 2021). In epidemiology, several studies using 
Google Trends data helped predict the spread of respiratory diseases in geographic regions of 
interest (Polgreen et al., 2008; Brownstein et al., 2009; Valdivia & Monge-Corella, 2010; 
Mavragani & Gkillas, 2020). Google Trends can also be used as a surveillance system to detect 
disease outbreaks. For example, Carneiro and Mylonakis (2009) argued that Google Trends 
could detect regional influence outbreaks 7-10 days before conventional surveillance systems 
used at the Centers for Disease Control and Prevention. In economics, Google Trends is used to 
predict several economic metrics, including unemployment claims in the US, Germany, and 
Israel (Choi & Varian 2009; Askitas & Zimmermann, 2010) and consumer and consumption 
sentiment (Huang & Penna, 2009; Schmidt & Vosen, 2009). 


Problem Statement 


The COVID-19 crisis brought immense challenges to educators and instructional staff 
around the globe. It is hypothesized that Instructional Design, Remote Learning, educational 
technology tools are critical knowledge and skills for ERT delivery. Therefore, Google and 
YouTube searches should reflect users’ information needs, especially at the beginning of the 
pandemic and the 2020-2021 school year. 


While models of information behavior exist to explain why users seek information, this 
exploratory study does not adhere to a specific information behavior model due to various users’ 
motivations, needs, and their roles in the search, and possibly avoidance, of information. Despite 
the lack of generalizability of information behavior captured in Google Trends, Case and Given 
(2016) identified the agreed-upon terminology in the information behavior literature, including 
information need, information seeking, and information behavior. An information need refers to 
the inadequate knowledge that users have. Information seeking is the conscious effort to acquire 
knowledge in response to the need or gap. Information behavior is complex and encompasses 
both conscious and unconscious information seeking, encountering, or avoidance. 
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Purpose & Significance of the Study 


The purpose of this study is two-fold. First, this study explores the popularity indices of 
COVID-19, Instructional Design, and Remote Learning as the primary search terms from Google 
and YouTube Search from January 2020 - April 2021 globally and in the United States. Second, 
the study seeks to identify the peaks and dips of user search interest in light of the COVID-19 
pandemic. The study explores the following research questions: 


@e RQ1: What are the related search terms for COVID-19, Instructional Design, and 
Remote Learning from Google Search and YouTube Search globally and in the USA? 

@ RQ2: What are the peaks and dips in popularity indices for COVID-19, Instructional 
Design, and Remote Learning from Google Search and YouTube Search globally and in 
the USA? 


The study’s findings allow for an exploration of search interest for knowledge in 
Instructional Design, Remote Learning, and educational technology tools to support the 
transition from face-to-face to the ERT format as educational institutions grapple with the 
pandemic. The following section describes the extraction methods and visualization components. 


Methods 


The Python Pytrends package was used to connect to the Google Trends API to extract 
search popularity indices of three keyword searches, COVID-19, Instructional Design, and 
Remote Learning, from Google Search and YouTube Search (Pytrends, n.d.). These search trend 
reports identify the normalized search volume of keywords and topics by state or province, 
country, and continent. Although Google Trends data does not indicate the real-time volume of 
searches, the data is normalized from 0 -100 to describe the search interest of topics (i.e., 0 
being the lowest and 100 being the maximum peak of interest) for the past five years or 2004 for 
specific search terms (FAQ about Google Trends Data - Trends Help, n.d.). 


For this particular study, the TrendReq, Interest Over Time, and Related Queries methods 
in the Pytrends Python package were implemented to automate the download of search trend 
reports from Google Search and YouTube Search from January 2020 - April 2021. Each report 
described the popularity indices of the keywords mentioned for the United States and globally. 
To further identify the related search term queries for COVID-19, Instructional Design, and 
Remote Learning, the Related Queries method in Pytrends allowed the extraction of the search 
trend data of related queries. Five known limitations of this study involved the following: 


1. Filtering certain search terms 

2. Extracting top search terms over rising search terms 

3. Eliminating related and repeated main search terms and their variations that do not align 
with the context of Instructional Design and Remote Learning 

4. Censorship of Google and YouTube in certain countries 

5. Global search volume can be skewed 
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First, Google Trends filters certain types of searches, including searches with a volume of 
0 or very few searches made by few people, duplicate searches, or repeated searches from the 
same person over a short period, and searches with special characters that use apostrophes and 
other special characters. Second, Google Trends allows extracting top and rising searches. While 
top searches are the terms with the most frequency, rising searches are the terms that have had 
significant growth over time. For this particular study, rising terms were not considered because 
of the changing nature of search volume growth. Unlike rising terms, top searches reflected the 
real search interest of individuals in the USA and globally. Third, the visualizations did not 
include related and repeated main search terms, including breathing exercises covid-19, covid-19 
vaccine, instructional materials design, and instructional design jobs. However, the terms above 
are included in Tables 2 and 3. Fourth, certain countries do not allow access to Google and 
YouTube. Crimea, Cuba, Iran, North Korea, and Syria do not allow access to Google services 
(Countries or Regions Where Google Workspace Is Available - Google Workspace Admin Help, 
n.d.). China, Eritrea, Iran, North Korea, Sudan, South Sudan, Syria, Tajikistan, and 
Turkmenistan do not allow access to YouTube services (Wikipedia contributors, n.d.). Fifth, 
there is a possibility that search index volume from Google and YouTube Search can be skewed 
based on a country’s search volume. For example, the majority of search volume may come from 
the USA and India for global search trends. For this study, the global search index includes the 
United States and other countries, excluding countries where Google and YouTube are banned. 


Two datasets were assembled to represent the popularity indices of various search terms 
to describe the search trends for the primary keyword searches and related terms in the United 
States and globally. Each dataset was subsequently visualized on a weekly basis in Tableau 
Desktop. Even though trend reports are insightful in understanding the peaks and dips of search 
interest of specific terms, the COVID-19 pandemic adds a vital context related to the information 
seeking of users in understanding the need for information in Instructional Design and Remote 
Learning. The design of the Tableau dashboards involved following six fundamental principles 
of information design by Tufte (1983), including comparison, causality, multivariate, integration, 
documentation, and context. Each Tableau dashboard is contextualized around COVID-19 
milestones and news related to vaccine development by the World Health Organization (WHO), 
U.S. Food Drug and Administration (FDA), United Nations (UN), and National Institutes of 
Health (NIH). These important milestones are annotated in the search trend data for the COVID- 
19 term along with its respective search popularity index. While the beginning of the 2020-2021 
school year varies in the USA, the school year was annotated on August Ist, 2020. A summary 
of milestones and indices is depicted in Table 1. 


Table 1 


Summary of Milestones and Search Popularity of Main Terms by Google and YouTube Search 


Milestone Global USA Global USA Global USA 
COVID19 COVID/ Instructional | Instructional Remote Remote 
Index COVID19 Design Design Learning Learning 
Index Index Index Index Index 


03/11/2020 Milestone 1: Google Google Google Google Google Google 
WHO declares 86 (Covid) 37 51 100 63 
COVID-19 a YouTube 100 YouTube YouTube YouTube YouTube 
pandemic 67 YouTube 57 51 63 63 
(Covid19) 
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06/26/2020 WHO announces 2 
billion doses of (Covid) 
COVID-19 vaccine 82 
YouTube 
(Covid19) 
12 


07/27/2020 Phase 3 clinical trial of Google 
an investigational (Covid) 
vaccine for COVID-19 73 
begin YouTube 
(Covid19) 
11 
08/01/2020 Milestone 2: Google 
Beginning of 2020 (Covid) 
school year (USA) 63 
YouTube 
(Covid19) 
iy 
12/18/2020 Milestone 3: Google 
FDA grants (Covid) 
emergency use 85 
authorization to Pfizer YouTube 
and BioTech’s (Covid19) 
coronavirus vaccine 35 
02/27/2021 Milestone 4: Google 
FDA grants (Covid) 
emergency use YouTube 67 YouTube 60 YouTube YouTube YouTube 
authorization to J&J's | 6 YouTube 60 15 15 
Covid-19 vaccine (Covid19) 
6 
04/20/2021 Milestone 5: Google Google Google Google Google Google 
Global tally of deaths 19 (Covid) 42 38 10 8 
from Covid-19 YouTube 53 YouTube YouTube YouTube YouTube 
surpasses 3 million 32 YouTube 42 42 10 14 
(Covid19) 
32 


Four Tableau dashboards were published to identify Google and YouTube search trends 
globally and in the United States for COVID-19, Instructional Design, and Remote Learning. 
The terms /nstructional Design, Remote Learning, and COVID-19 are selected by default in the 
dashboards. Also, the terms of interest contained annotations of the selected terms with their 
respective peaks of search interest and their average popularity indices. Multiple terms can be 
selected and highlighted using the selection menu to compare search trend data further. As 
search terms are chosen, the terms are color-coded with their respective labels on the trend chart. 
To better navigate the charts, it is recommended to view the dashboards in full-screen mode. 


Findings and Discussion 


It is essential to point out that Covid/9 had more peaks in Google Search globally than 
Covid-19. In the United States, covid was more prominent than other term variations (e.g., 
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Covid-19, Covid 19, and Covid19) in Google Search. In the United States, Covid was the 
primary search term in Google Search, whereas Covid/9 was the prevalent search term in 
YouTube Search. Interestingly, the term Emergency Remote Teaching and its variations (e.g., 
ERT, Emergency Remote Learning, and Emergency Remote Teaching and Learning) did not 
appear as related terms to /nstructional Design or Remote Learning. It seems that users were 
more acquainted with Remote Learning and associated educational technology tools. 


RQ 1: What are the related search terms for COVID-19, Instructional Design, and Remote 
Learning from Google Search and YouTube Search globally and in the USA? 


For COVID-19, the related terms included Covid/9 for global Google Search and Covid 
for USA Google Search. Also, related search terms (i.e., cases, deaths, news, symptoms, and 
testing) to Covid19 were prevalent in Google Search in the United States. Related combinations 
such as Instructional Design + Remote Learning were present in both global and USA Google 
Search. Remote Learning also contained related terms, including E-Learning + Remote Learning 
in global Google Search and covid19 + remote learning in the USA Google Search. 


Interestingly, related terms to Remote Learning in global Google Search included an 
educational technology provider (i.e., Seesaw Remote Learning) and a web conferencing tool 
(1.e., Zoom). Seesaw Remote Learning is an online classroom app that delivers asynchronous and 
synchronous learning with feature-rich communication and class management tools. In addition, 
Seesaw provides integration of Google Classroom and support of BYOD (i.e., Bring Your Own 
Device) to allow students’ personal devices with access to school resources (Seesaw, n.d.). Zoom 
is better suited for synchronous remote and hybrid learning as a web conferencing tool for 
teaching and learning. Zoom also supports a variety of engagement strategies, including 
formative strategies (e.g., quizzes and surveys) and brainstorming activities (e.g., breakout 
rooms, screen sharing, and interactive whiteboard) (Zoom for Education, n.d.). Table 2 
summarizes the related terms or queries for the search terms of interest (COVID-19, Instructional 
Design, and Remote Learning) from Global and USA Google Search. 


Table 2 


Related Search Terms for Global and USA Google Search 
COVID-19 Instructional Design Remote Learning 


Covid-19 Covid Instructional Design } Instructional E-Learning + Doe Remote Learning 
Covid19 Covid 19 Instructional Design | Design Remote Learning Zoom 
covid19 + remote Covid-19 + Remote Learning Instructional Elearning + Remote | Seesaw Remote 
learning Covid19 Design + Remote | Learning Learning 

Vaccine Learning Zoom Zoom Remote Learning 


Covid19 Cases Seesaw Remote Zoom For Remote 
Covid19 Deaths Learning Learning 
Covid19 News School Remote What is Remote 
Covid Symptoms Learning Learning 
Covid19 Us Remote Learning Tips 
Covid19 Testing Remote Learning 
Setup 
Remote Learning Fall 
2020 
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remote learning 
covid19 + remote 
learning 


For the term Covid-19, the related terms were variations of the search term (covid and 
covid 19), covid-19 vaccine, and covid-19 + remote learning in global Youtube Search. In 
contrast, in USA YouTube Search, related terms were covid and breathing exercises covid-19. In 
terms of Instructional Design in global YouTube Search, related terms were related to the design 
of instructional materials and jobs in Instructional Design. However, related terms to 
Instructional Design were not present in the USA YouTube Search. For Remote Learning in 
global YouTube Search, three related terms were related to covid and two YouTube channels 
that host free educational content, including Vidura and Gours. Vidura eLearning is based in Sri 
Lanka with approximately 19,000 subscribers and provides educational content from grades | 
through 5 (Vidura eLearning, n.d.). Gours eLearning is based in India with about 52,000 
subscribers and offers financial education and certification resources in online instructor-led and 
self-paced formats (Gour’s e-Learning, n.d.). Table 3 summarizes the related terms or queries for 
the primary key search terms (COVID-19, Instructional Design, and Remote Learning) from 
Global and USA YouTube Search. 


Table 3 


Related Search Terms for Global and USA YouTube Search 


COVID-19 Instructional Design Remote Learning 
a 


covid covid instructional N/A covid 

covid-19 vaccine breathing exercises materials design vidura elearning 
covid 19 covid-19 instructional design gours elearning 
covid-19 + remote jobs 

learning 


RQ 2: What are the peaks and dips in popularity indices for COVID-19, Instructional 
Design, and Remote Learning from Google Search and YouTube Search globally and in the 
USA? 


A total of ten findings were found in comparing popularity search indices for Google and 
YouTube Search in the United States and globally. Overall, this study’s terms of interest and 
related terms exhibited similar peaks and dips of search interest at the start of the pandemic in 
March 2020 and the 2020-2021 school year. The section below highlights the findings by global 
and USA Google Search and YouTube Search. 


Global Google Search 


Finding #1. The search term Covid/9 had more peaks than Covid-19. Covid19 received 
more search interest than Covid-19 in Google Search globally. 
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Finding #2. The search terms Covid19, E-Learning + Remote Learning, Elearning + 
Remote Learning, Instructional Design + Remote Learning, and Remote Learning Fall 2020 
exhibited similar peaks of interest in Google Search globally. 


Finding #3. Remote Learning had two significant peaks of interest at the beginning of the 
pandemic and the 2020-2021 school year, followed by a downward trend in Google Search 
globally. 


USA Google Search 


Finding #4. The term Covid was the most prominent with an upward trend than any other 
variation of the term (Covid-19, Covid 19, and Covid19) in USA Google Search. 


Finding #5. The term /nstructional Design was at its highest peak (100) on July 12th, 
2020. Then it levelled off in 2021 with search interest approximately at 65 in USA Google 
Search. Jnstructional Design remained independent of the pandemic’s peaks of interest for Covid 
and its related terms. 


Finding #6. The terms Covid and Remote Learning terms exhibited similar peaks of 
interest at the beginning of the pandemic and the 2020-2021 school year. About Covid, Remote 
Learning was at its highest after 18 days of the beginning of the pandemic and after 32 days at 
the beginning of the school year in August 2020. After October 2020, the Remote Learning term 
experienced a drastic downward trend by the end of 2020 and the beginning of 2021. It would be 
possible that public and private institutions have leveraged educational technologies to deliver 
learning in online and hybrid formats at the beginning of the Fall of 2020. 


Global YouTube Search 


Finding #7. The terms E-Learning + Remote Learning, Elearning + Remote Learning, 
Remote Learning, and remote learning had similar peaks and dips search interest in YouTube 
Search globally. 


Finding #8. Remote Learning, remote learning, and Instructional Design + Remote 
Learning experienced two significant peaks at the height of the pandemic and the 2020-2021 
school year, followed by a downward trend in early 2021. Global YouTube Search interest for 
Instructional Design did not seem to show significant peaks and dips in 2020 and early in 2021. 


USA YouTube Search 

Finding #9. In the USA YouTube Search, Zoom, Zoom Remote Learning, Covid19, 
Covid-19, Instructional Design + Remote Learning, and covid19 + remote learning exhibited 
significant peaks at the beginning of the pandemic in March 2020 and the 2020-2021 school year 
in August 2020. 


Finding #10. In the USA YouTube Search, /nstructional Design + Remote Learning and 
Zoom showed similar search interest with peaks of interest at the beginning of the pandemic and 
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2020-2021 school year. However, Instructional Design exhibited no significant peaks or dips in 
search interest on YouTube and experienced two significant peaks (100% interest) on July 12th 
and September 20th, 2020. 


Implications & Future Research 


The study presents implications for practice and research. The significance for 
practitioners involves bringing awareness of centralized information hubs to help educators and 
students transition to ERT and establishing communication plans for resource-sharing among 
Instructional Designers and Technologists who support educators’ professional development. 
The implication for researchers identifies the need to study the main search terms and their 
related queries to understand users’ information seeking and needs in emergency situations. 
Research can also focus on anticipating information seeking by exploring upward and downward 
trends in search queries. 


Conclusion 


This study accomplishes two goals: (1) keeping an archive of information seeking from 
search engines and (2) highlighting the search interest of Instructional Design, Remote Learning, 
and educational technology globally and in the USA across Google and YouTube Search from 
January 2020 - April 2021. The search term Remote Learning and Covid had similar peaks of 
search interest in USA Google Search at the beginning of the pandemic in March 2020 and the 
2020-2021 school year. In USA Google Search, the highest peak for Remote Learning occurred 
18 days after the highest peak for Covid, and 32 days after the beginning of the school year, 
given the assumption that the school year began in the first week of August. 


The search term variations of the COVID-19 terms were Covid-19, Covid19, and covid19 
+ remote learning in global Google Search. Related terms to COVID-19 included Covid19, 
Covid19 Cases, Covid19 Deaths, Covid19 News, Covid Symptoms, Covid19 US, and Covid19 
Testing in USA Google Search. Related terms to /nstructional Design were Instructional Design 
+ Remote Learning in USA Google Search. Associated terms for Remote Learning showed 
educational technologies (e.g., Zoom, SeeSaw, Remote Learning Setup, and E-Learning) in 
global and USA Google Search. In USA Google Search, related terms to Remote Learning 
included governmental guidance (e.g., Doe Remote Learning) at the beginning of the 2020-2021 
school year. 


The related search terms for COVID-19 were vaccine and remote learning in global 
YouTube Search, whereas breathing exercises covid-19 was present in USA YouTube Search. 
Instructional Design did not have any related terms to COVID-19 and Remote Learning in global 
and USA YouTube Search, except for instructional materials design and instructional design 
jobs. In global YouTube search, related terms to Remote Learning included educational channels 
that target K-12 students and professional development resources. However, no associated terms 
for Remote Learning were present in USA YouTube Search. 
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As educational institutions face last-minute adaptations of face-to-face curricula to online 
and hybrid formats, the COVID-19 pandemic has accelerated the need for knowledge related to 
Instructional Design, Remote Learning, and educational technology tools. 
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Abstract: Compared with traditional classroom teaching observation, the classroom behavior 
analysis system using information technologies can effectively improve the validity and accuracy 
of interpretation of classroom teaching behaviors. With the development of information 
technology, the combination of the application of teaching methods and artificial intelligence has 
become an inevitable trend. Based on the S-T teaching behavior analysis method, an intelligent 
classroom teaching behavior analysis system using multi-modal recognition technology was 
proposed, which applied various information technologies to collect classroom teaching behavior 
data and code them automatically. The accuracy and reliability of the presented system was 
verified through the collection and analysis of a large number of classroom behavior data, which 
provided a valuable scientific analysis tool for classroom teaching behavior research. 

Keywords: Classroom teaching behavior analysis, S-T analysis method, Multi-modal recognition 
technology 


1 Introduction 


Today, as education informatization is gradually deepening, the theory of teaching and 
learning continues to develop. Gradually integrate new educational concepts and technical means 
into classroom teaching, and traditional teaching evaluation methods have gradually failed to meet 
the needs of teaching development. Classroom teaching evaluation methods need to keep pace 
with the times, classroom teaching practice calls for in-depth study of classroom teaching behavior 
(Taut & Sun, 2014). This study completed an intelligent analysis system based on S-T analysis 
method and multi-modal recognition technology, and verified its accuracy and reliability. This not 
only provides a basis for teachers to improve teaching content and teaching methods, but also has 
great significance for promoting the development of teachers' professional ability. 


2 Literature Review 
2.1 the S-T Analysis Method 

The advantages and disadvantages of classroom teaching are largely related to the interaction 
between teachers and students. Classroom is the main place where teaching activities take place. 


Classroom behavior takes place in this space and the information transmission between teachers 
and students interacts and interacts in this space. As the main body of the teaching process, the 
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behavior of students in the classroom can directly reflect their learning status. As a guide for 
students to learn, teachers also play an important role in guiding behavior. Therefore, analysis of 
classroom teacher and student behavior is conducive to analyzing classroom teaching process. 
Some scholars have been working on software for collecting and analyzing classroom teaching 
behavior data and have made good progress (Yun et al. , 2018). Some scholars have conducted 
classroom observations from multiple angles and at multiple levels, and also provided new ideas 
for data collection and processing of this study (Xie & Cao, 2010). Japanese scholars first proposed 
the S-T teaching analysis method for classroom teaching analysis. S-T analysis method is the 
Student-Teacher analysis method, which is a teaching analysis method that visually expresses the 
teaching personality in a graphical way (Jun & Ou, 2011). The main steps of the method are 
divided into data sampling, analysis of time series data, drawing S-T diagram, and drawing RT- 
CH diagram to judge the teaching mode analysis. In the actual classroom teaching process, the 
specific manifestations of Teacher’s behavior and Student’s behavior are shown in Table 1. Since 
the Student’s behavior includes all behaviors other than the Teacher’s behavior, in addition to the 
11 behaviors described in Table 1, it can be classified as “Others” as a form of Student’s behavior. 


Table 1: Specific manifestation of T behavior and S behavior 


Categor | Numbe | Manifestations Category | Numbe | Manifestations 
y r r 
Teacher | 1 explanation Student’s | 7 Student's speech 
s 2 demonstration behavior | 8 Student’s thinking, 
behavio calculation 
r 3 blackboard 9 Student taking 
notes 
4 using various 10 Doing experiments 
media to prompt or finishing 
homework 
5 question and 11 Keeping silence 
name 
6 evaluation and 12 Others 
feedback 


This study draws on the traditional classroom teaching behavior analysis method, and uses a 
variety of information technologies to collect and analyze classroom teaching behaviors. It will 
record actual classroom teaching behaviors, help researchers and teachers analyze classroom 
teaching behavior, and promote professional development of teachers. 


2.2 Kinect System Principle 


In 2010 and 2014, Microsoft released KinectV1 and V2 successively. KinectV2 includes a 
color camera, depth camera, infrared light emitter and microphone, which can realize real-time 
dynamic capture, image recognition, microphone input, voice recognition, social interaction and 
other functions. . The official system of Kinect, Kinect for Windows SDK, contains drivers, 
original sensing data flow development interface, user interface, installation file data, and can be 
used for secondary development. KinectV2 uses TOF technology (Time of Flight) to obtain depth 
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image information by calculating the projected infrared rays and the return time after reflection, 
and then segment the human body from the background image through the machine learning 
algorithm, and then estimate the three-dimensional coordinate information of the human body joint 
points (Roque et al. ,2019). It is less affected by the environment and light. It gets rid of the high 
requirements of traditional motion capture technology in experimental environment, experimental 
equipment, and the accuracy of markings. It can capture user actions, facial expressions and voice 
sequences in real time for the purpose of machine interaction. Skeleton tracking technology is the 
core function of Kinect. This technology uses deep vision technology and uses complex algorithms 
such as machine learning and matrix changes to determine the three-dimensional coordinates of 
joint points. 

Skeleton tracking technology is the core function of Kinect. The deep vision technology 
adopted by this technology uses complex algorithms such as machine learning and matrix changes 
to determine the three-dimensional coordinates of the joint points. The working process 
includes(He & Li 2020): 

()Human contour segmentation: Kinect obtains the depth image information through the depth 
sensor, and through edge detection, Noise threshold processing and other technologies separate 
the human target (“T”-shaped object, which will be recognized as a human body by Kinect) from 
the environmental background to obtain a depth image of the human body; 

(2)Human body part recognition: the human body separated from the depth image Different parts 
of the human body are identified in the contour, and a large amount of data is used to train and 
classify through the classification algorithm of the decision forest. Each pixel is labeled with a 
category label and classification probability to classify various parts of the human body, such as 
the head and shoulders; 

(3)Joint point positioning: After completing the recognition of human body parts, Kinect analyzes 
all aspects of the human body and uses machine learning algorithms for joint positioning. 

In traditional sports teaching and training, the teaching method is one-to-many, and students' 
understanding and learning progress are uneven, resulting in poor teaching or training effects. In 
order to solve these problems, more and more professional sports training began to apply motion 
capture to sports training and teaching. Kinect determines the accuracy of the exerciser's 
movements by accurately grasping the contour and position of the exerciser's body, and performs 
corresponding training or operation according to the movement function. The trainer's technical 
movements are identified through the Kinect device. After the computer processes the collected 
data, it makes correct judgments on the technical movements and feeds them back to the 
practitioners to encourage them to practice repeatedly, correct wrong movements and improve 
teaching or training efficiency. 

The system analysis results show that there are 22 researches on the development and 
application of Kinect in auxiliary physical education and training; these researches involve 
competitive events (badminton, basketball, etc.), flexibility events (yoga, Tai Chi, etc.). Kinect's 
process characteristics as an auxiliary teaching and training tool are: 

(4) Use Kinect-based system equipment to capture human bones and depth information; 

(2) Using self-occluded joint point information restoration algorithms and filters to collect depth 
images or motion information of each joint Process; 

(3) Compare and analyze the collected information with the pre-set standard actions; 

(4) Provide real-time feedback or evaluation to inform the trainer how to adjust the actions and 
posture, improve the actions in time, and continue training. 
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3 Model building 


With the development of computer science and precision manufacturing, the automation of 
video-based classroom teaching observation has become an inevitable trend. Multi-modal 
recognition technology uses a variety of sensory recognition techniques such as depth image and 
phonetic intonation to perform behavior recognition. This will help to further improve classroom 
information collection efficiency, coding efficiency, information classification accuracy, and 
greatly enhance teachers' interpretation of classroom teaching effects. Therefore, in order to realize 
this idea, this study collected classroom teaching behavior data through the various information 
technologies, such as depth image analysis technology, human skeleton tracking technology, 
speech analysis technology and so on. 

S-T analysis method is an analytical method that expresses the character of teaching in a 
graphical way. This analysis method divides the behaviors into two categories: student (S) 
behavior and teacher (T) behavior, which reduces the ambiguity of behavior classification in the 
teaching process and increases the objectivity. For the recorded behavior sequence data, it can 
calculate the T behavior occupancy rate (Rt), the behavior conversion rate (Ch), and plot the Rt- 
Ch diagram (Jun & Ou, 2011). According to this, the teaching mode adopted in the class can be 
judged, and the teaching method can be improved by using a visual method. It does not require 
other complicated calculations, and it is very convenient to use, which is conducive to promotion 
and implementation. 

The depth image analysis technology is used to recognize the objects in the visual filed and 
analysis the geometric characteristics of objects (Yeloglu et al. , 2015). This technology improves 
the accuracy of computer image recognition and promotes its development. The human skeleton 
tracking technology is to recognize the joints of human skeleton and collect the coordinate data of 
human skeleton. Some scholars use the human skeleton tracking technology in the rehabilitation 
measurement of disabled people, determine and improve the accuracy of the system’s joint 
tracking, which achieves good results (Mobini et al. , 2014). Some scholars use the human skeleton 
tracking technology to apply the somatosensory interaction system to multimedia teaching in the 
classroom environment (Sommool et al. , 2013). The speech analysis technology is to recognize 
the speech features of human, which has been used in many human-computer interaction systems. 

Based on the above analysis, this study constructs a classroom teaching behavior analysis 
model based on S-T analysis method and multi-modal recognition technology, as shown in Figure 
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Figure 1 Classroom behavior analysis model 
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Through the depth image recognition technology and the human skeleton tracking 
technology, combined with the speech recognition technology, it can collect all the data such as 
posture and facial behavior of teachers. At the same time, the model combines with the S-T 
analysis method to encode the teacher behavior data, so as to comprehensively collect the teachers' 
status, to establish a low-cost, real-time classroom state detection system that does not affect the 
original listening state of the classroom. 

The system software architecture based on S-T analysis method is mainly divided into display 
layer, business layer and data layer. The display layer is mainly used for the display of the interface 
and the interaction with the user, mainly using the Microsoft .NET Framework form application 
technology. On the interface, users can choose to run programs, draw charts, view statistics and 
view data sources. It is a simple and effective operating system for users. The business layer mainly 
implements the functions of data processing and chart drawing. It mainly uses the .NET 
Framework form application and Windows.Forms.DataVisualization. Charting to realize the chart 
drawing. It is a layer of internal logic implemented throughout the software. The data layer 
implements the function of storing data and reading data by the user, and uses a text file to store 
data, as shown in Figure 2. The design idea of the whole system is to complete the coding 
information of the program design by comparing the continuous changes between every 50 frames. 


Upper . 
application 
Data layer Data storage 


Sensor Depth image | |Skeletal image|| Color image |Voice intonatio1 


Figure 2 System software architecture diagram 


4 Method 
4.1 Participants 


In this study, two teachers from a university in East China are used as experimental subjects 
to carry out experiments, and the course of "Human-Computer Interaction" in Educational 
Information Technology is selected as the research sample. The specific unit course selected is a 
practical course of human-computer interaction. The content of the course mainly guides students 
to use six degrees of freedom robotic arms. The course of this unit is relatively difficult, and 
students need to be guided to complete the course study, program writing, and exchange summary 
independently. 
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4.2 Experiment Procedure 


The experiment is carried out on the basis of ensuring that the normal classroom order of 
teachers and students will not be affected. Before the course starts, first of all, arrange the 
equipment to ensure that the depth sensor camera and high-definition camera can record all the 
actions of the teacher in the classroom. Monitor whether the equipment is operating normally 
during the course. Then distribute the course videos to three researchers, and manually code the 
teacher behavior. Finally, the manual coding result is compared with the system coding. The 
specific classroom content is required to be designed to be diversified, including at least one of the 
codes in the S-T teaching analysis method, and to ensure a certain proportion of classroom 
interaction design. 


5 Results 


The system encodes and analyzes classroom behavior in a frequency of 5 seconds. In order 
to detect the reliability of the system, after the automatic coding is completed by the intelligent 
teaching behavior analysis system, the experiment initiator organizes the experimenter to complete 
the video-based classroom coding analysis, and finally compares their analysis results. The results 
show that the coincidence degree between the intelligent analysis system and the manual coding 
method reaches 86%, which verifies the feasibility and effectiveness of the presented intelligent 
system. 

In order to increase the user's intuitive perception of the time dimension of classroom teaching 
behavior, this study visualized one course’s classroom data according to time while ensuring 
accuracy. The broken line in Figure 3 not only reflects the changes in teachers’ classroom behavior. 
The abscissa is time in seconds. 1-6 respectively indicate different classroom teaching behaviors, 
namely explanation, demonstration, blackboard, using various media to prompt, question and 
name, evaluation and feedback. From the overall behavior data, the teacher's classroom behavior 
corresponding to the data in Figure 3 has the largest proportion of explanations, indicating that the 
teacher has more explanation time in the classroom. 
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Figure 3 Distributions of classroom teaching behavior 
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6 Conclusion and Discussion 


Researchers and teachers use the proposed intelligent classroom teaching behavior analysis 
system based on S-T method and multi-modal recognition technology to analyze teaching 
activities in the classroom, which will improve the quality of classroom teaching and learning 
effectively, be helpful to the professional development of teachers, and students can also benefit a 
great deal. This study explored the classroom teaching behavior model, designed and developed 
the intelligent analysis system to reduce the difficulty of classroom teaching behavior observation, 
improve the efficiency and quality of classroom teaching activities. It is of great significance to 
improve the quality of education. 

This research has achieved multi-modal classroom behavior recognition, but the research 
currently only uses two teachers as samples for experiments. When it is extended to more teachers, 
different behaviors of different teachers need to be considered to improve the accuracy of 
classroom behavior recognition degree. Future research will add students’ classroom behaviors to 
form a complete multi-modal classroom behavior recognition system for teachers and students. 
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The COVID-19 pandemic quickly and drastically changed the field of education in 2020. 
Teachers faced challenges when confined to their homes and had to begin emergency remote 
teaching with their students. It is important to recognize what these teachers experienced and 
find ways to prepare for a crisis-prompted teaching situation in the future. The purpose of this 
phenomenological study was to explore the lived experiences of Pennsylvania elementary 
classroom teachers who engaged in emergency remote teaching during the COVID-19 pandemic. 
Semi-structured Zoom interviews of eight elementary teachers who participated in emergency 
remote teaching provided data for qualitative phenomenological analysis. Findings revealed five 
themes: Emotional stress with the transition to emergency remote teaching, technological and 
instructional barriers, changes in content, pedagogy and technology use, increased workload and 
self-reliance, and positive relationships with students and colleagues. Mishra and Koehler’s 
(2006) technological, pedagogical, and content knowledge (TPACK) framework was used to 
guide the data collection and discussion of the findings in this study. The practical implications 
of this study include training and policies related to pandemic or emergency teaching planning. 


Keywords: COVID-19, Emergency Remote Teaching, Teachers 


INTRODUCTION 


In December 2019, a novel coronavirus was identified in a virus outbreak in Wuhan, China 
(Centers for Disease Control and Prevention [CDC], 2020a; CDC, 2020b). Schools across the 
nation closed in order to prevent the spread of this novel coronavirus. As early as late February 
2020, superintendents and governors closed schools across the United States, first temporarily 
and then extending to the rest of the academic year, affecting at least 50.8 million public school 
students (Decker et al., 2020). On March 13, 2020, Pennsylvania Governor Tom Wolf ordered 
the closure of all schools in Pennsylvania for two weeks, a measure aimed at slowing the spread 
of the COVID-19 ("Governor Tom Wolf," 2020). School teachers in Pennsylvania had to find 
an alternate means of providing education to their students, turning to emergency remote 
instruction for the remainder of the spring of 2020. Many teachers found themselves unprepared 
for the challenges they faced (Hodges et al., 2020). 
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About COVID-19 


Coronaviruses are a large family of viruses that may cause illness in animals or humans 
(WHO, 2020). In humans, coronaviruses cause respiratory diseases as simple as the common 
cold or as complex as severe diseases such as Middle East Respiratory Syndrome (MERS) and 
Severe Acute Respiratory Syndrome (SARS) (Centers for Disease Control and Prevention 
[CDC], 2020a; WHO, 2020). COVID-19 can be spread from person to person via small droplets 
in the nose or mouth when a person coughs or exhales (CDC, 2020a; WHO, 2020). On February 
11, 2020, the World Health Organization (WHO) announced the new coronavirus’s official 
name, using their best practice to name new human infectious diseases (CDC, 2020a). The CO 
stands for corona, the VI for the virus, the D for disease, and -19 for the year when the outbreak 
began, 2019 (CDC, 2020a). On March 11, 2020, the WHO declared the novel coronavirus 
outbreak a global pandemic (Cucinotta & Vanelli, 2020). The potential for spread led to social 
distancing and mask use across the country. 


COVID-19 and Emergency Remote Teaching 


Remote teaching, which is different from online teaching, allows educators to teach with 
technology to ensure students’ continuity of learning at a distance (Trust & Whalen, 2020). 
Remote teaching occurs “during emergencies when teachers and students are not accustomed to 
using online platforms or technology to deliver instruction” (Russell, 2020). Since March 2020, 
emergency remote teaching has become the terminology used to refer to this teaching type 
during the pandemic (Craig, 2020; Hodges et al., 2020). 

As aresult of emergency remote teaching, teachers experienced an unprecedented and 
unplanned disruption to education that changed the work of many teachers suddenly and in many 
aspects (Kaden, 2020). Teachers around the world "frantically and emergently switched to 
remote teaching" (Schlesselman, 2020). Teachers were given anywhere from a few hours to a 
few weeks to prepare to move all instruction to a remote format (Gacs et al., 2020) and expected 
to become online teachers overnight (Gudmundsdottir & Hathaway, 2020). Teachers were 
suddenly confined to working from home, removed from students, and challenged to learn new 
technologies (Kaden, 2020). 

When teachers begin instructing in an online environment, they generally have time to 
meet their needs before beginning (Gacs et al., 2020; Moore-Adams et al., 2016). Many teachers 
who experienced emergency remote teaching did not have time to prepare or receive the 
professional development necessary to begin such a massive undertaking (Paesani, 2020). With 
this sudden transition to emergency remote teaching, educators faced an abnormal environment 
where they had to get ready quickly, without the help commonly available and access to quality 
resources, all while confined to their homes (Huang et al., 2020; Wake et al., 2020). Teachers 
had to use whatever resources they had access to at the time without any clearly defined 
standards or understanding of online teaching (Schlesselmen, 2020). This move to emergency 
remote teaching resulted in an increased workload for teachers, challenging them to provide 
meaningful educational experiences to their students, despite what was happening around them 
(Kaden, 2020). 

The change to emergency remote teaching happened so fast that it is essential to pause 
and reflect on those experiences (Pacheco, 2020). The emergency remote teaching situation 
experienced by educators in the spring of 2020 offered many "lessons on preparedness that can 
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and should be addressed in anticipation of another crisis teaching situation" (Pacheco, 2020). The 
COVID-19 pandemic completely disrupted the education system and forced a wake-up call to 
strengthen public education (Kaden, 2020). Teachers need to be prepared to teach entirely online 
and immediately shift to emergency remote teaching (Major, 2020). Teachers will need tools and 
guidance should the need for emergency remote teaching arise in the future (Gacs et al., 2020). 


METHODOLOGY 
This study employed a qualitative design using a phenomenological approach. 
Participants 


Interview data were collected from eight teachers at various public schools in 
Pennsylvania. The participants were recruited from different schools and elementary grade levels 
(kindergarten through sixth) to get stories from multiple contexts, richer data, and avoid 
homogenizing their experiences. A purposeful sampling strategy was used and the criteria for 
selection of the participants included: (i) being elementary classroom teachers in Pennsylvania 
who transitioned to emergency remote instruction due to the COVID-19 pandemic, (ii) having no 
prior experience with online teaching or emergency remote teaching, (iii) being self-contained 
elementary classroom teachers. Eight participants were purposefully selected to participate in the 
study using these criteria. 


Instrumentation 


The interview protocol consisted of 14 semi-structured questions. The interview protocol 
was created based on Brinkmann and Kvale’s (2015) method of preparing for the interview by 
setting the stage and working off a script, yet moving through the interview like a conversation 
between “two partners about a theme of mutual interest” (p. 149). The interview protocol was 
designed using topics and questions related to the focus of the study. The interview protocol 
contained questions aimed at gathering a detailed description of the participants' experiences 
with the phenomenon. This approach is supported by Merriam and Tisdell’s (2016) findings that 
interviewing is the primary method of data collection to arrive at the essence of a lived 
phenomenon. The questions asked in the interview protocol were based on the work of 
Brinkmann and Kvale (2015), who emphasize the importance of attempting to understand the 
world from the participants’ point of view. 


Data Collection and Analysis 


Interviews were conducted via Zoom due to ongoing safety concerns related to social 
distancing and the COVID-19 pandemic. The Zoom interviews were recorded using the 
application; the Zoom application allows users to record audio and save it right to the computer 
in an M4A format. M4A is an extension that signifies an audio recording. Each interview lasted 
between 30 and 50 minutes. They were transcribed using the website Otter, a service that turns 
voice conversations into transcribed notes. The transcripts were reread for precision twice and 
then sent individually to each participant to check them for accuracy and report any 
discrepancies. Having the participants review the transcripts was done to increase the reliability 
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and validity of the data. The transcripts were then coded using Dedoose. The data were analyzed 
based on code frequency, key terms, and themes across all interviews. Pseudonyms were used 
for confidentiality. 


RESULTS 


The analysis of the interviews revealed five themes resulting from the categorization of 
codes: emotional stress with the transition to emergency remote teaching, technological and 
instructional barriers with emergency remote teaching, changes in content, pedagogy and 
technology due to emergency remote teaching, increased workload and self-reliance with 
emergency remote teaching, and positive relationships during emergency remote teaching. 


Emotional Stress 


The study revealed that no one expected a worldwide pandemic and did not predict the 
resulting shift in education. Teachers had difficulty grasping the notion of what was happening in 
the world. They experienced stress when setting up their new virtual classrooms and working 
from home. 


Fay: I cried. Like cried. I cried because, you know... And then, I don’t know if we had a week to 
prepare or try to figure it all out. But no, I think I just went through a realm of emotions, you 
know. First of all, we’re scared. We don’t know what’s going on. 


Anne: Like just teacher burnout, we worked all day and then all night. We were always willing 
to do it, but it led to extremely long, long, long days. I have children I need to take care of, so it 
was hard balancing my family and being a teacher and supporting all of their families at the same 
time. 


Technological and Instructional Barriers 


This study revealed that all teachers experienced difficulty with device access and 
connectivity. The teachers in this study were without their materials and had issues with students 
attending online classes and completing their work. 


Anne: We are a country school. Some of our children’s homes don’t have cable TV running to 
their properties. So, to get a hotspot to work in a wooded area or where a mountain is blocking 
you, sometimes they were very spotty. So, we had a lot of messages that they were behind 
because it was difficult for them to complete the assignments with the lagging. 


Jane: It’s obviously not where I wanted it to be. Because it wasn’t direct, I wasn’t in person with 
my students. I did the best I could with what I had to work with. Knowing I didn’t have access to 
any of my materials at school, so I was able to just take my thoughts and put them into writing 
and pass that along to the parents and hope that, you know, they had things at home that they 
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could utilize to practice those skills. After they left on March 13">I had no interaction or contact 
with my students at all. 


Changes in Content, Pedagogy, and Technology 


The teachers in this study all experienced changes and adaptations to content in some 
way. The teachers either changed the content they were used to teaching or taught the same 
content but with a different effort. The teachers all reported learning how to use new technology 
and then altering their content and practices to work with the new technology use. 


Anne: A colleague of mine mentioned Loom, so I started recording myself teaching the lessons. 
The students would have direct links to me teaching them on a recording. 90% of it was 
asynchronous. I would hold a Zoom every day. They had to sit and listen from their dining room 
tables instead of the classroom. 


Lila: | saw a lot of creativity come out of kids that I don’t know if I would have seen had we not 
gone remote. I really did see some kids do some projects that I felt like, wow, they put forth a lot 
of time and effort into this. I got to see a different side of the kids in that manner. 


Increased Workload and Self Reliance 


All participants found themselves with a new, more intense workload in addition to 
needing to rely on their ability to provide instruction to students remotely since the time and 
resources for professional development were not perfect for this situation. 


Sue: They kept on throwing things at us. Like, you have to do this. You have to document that. 
And there was not much guidance; it was just a list of to-dos without how to do them. And it 
worked out wonderfully, but there were really high demands on us. I truly felt like I was flying 
the plane as I was building it. Each day I would learn something new, or something would go 
wrong. And I would have to learn on the fly how to fix it. I did not have my materials. I was 
learning not only how to do Zoom and teach material online, but I was home as well. 


Nancy: Ona Monday and a Friday, I had two different meetings because of working schedules 
and stuff like that. During the day, we were required to be there for almost like office hours. We 
needed to respond to a parent. I spent a lot of time creating different content for the kiddos. 


Positive Relationships 


Each of the eight participants noted positive relationships with either colleagues, 
students, or both. Some even found positive relationships with administration or students’ 
parents during the pandemic. The participants found that the circumstances allowed them to get 
to know their students on a level that may not have been possible without the pandemic. The 
reliance on their coworkers to get through this unprecedented time also strengthened their 
relationships with each other. 
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Fay: The students really just wanted to talk to us. And I think we were told that’s pretty much 
okay. Especially for eight-year-olds, you know, it was just to check in. They most wanted us like 
making sure the kids were okay. Not like their safety but you know like, mentally okay. I would 
do one on one meetings with them. And that didn’t really work. Because, again, they just want to 
show you, their cat. And I’m like, okay, let’s talk about your math. 


Pam: We have like a very close group with our special ed teachers. So, we’re constantly in 
group chats together; we opened like Google Classrooms to communicate, share ideas, like if we 
wanted to use the same thing in our classroom. So just the ongoing communication and 
collaboration, we always share ideas with each other, which is helpful. 


DISCUSSION AND CONCLUSION 
Discussion 


This study’s findings provided an understanding of the lived experiences of elementary 
teachers who participated in emergency remote teaching during the COVID-19 pandemic. The 
analysis of the interviews uncovered five themes related to these experiences. 

Each participant described the changes they experienced in their teaching as a result of 
emergency remote teaching. 

All participants noted emotional stress with the transition to emergency remote teaching; 
they felt fear, frustration, and uncertainty. The teachers were not prepared to become emergency 
remote teachers. Even the teachers with advanced technological skills reported they never 
expected a worldwide pandemic to change how they provided education to their students. 
Teachers were given little to no notice that everyone would be sent home to learn, at first for two 
weeks and then for the remainder of the 2019-2020 school year. The teachers did not have time 
to prepare and gather all the materials they would need to instruct from home and found this time 
very stressful. Teachers reported they did not get clear directions from their administrators, who 
were also trying to figure out what was going on and what to do. 

Each of the participants noted numerous issues with technological and instructional 
barriers when conducting emergency remote teaching. Teachers faced technological barriers 
right away because not all students had a device at home. Many districts worked to get devices to 
students, but this took time. Some teachers were surprised, but students lacked access at home, 
so districts had to provide things like hot spots to connect to the Internet. The schools in rural 
areas had consistent connectivity problems, with students reporting glitchy connections or lack 
of connection in bad weather. The teachers had to figure out how to use these platforms and apps 
and then teach the students, who were also unfamiliar. As emergency remote teaching continued, 
teachers encountered consistent technology problems and solved them as they came up. 

They also faced changes that had to be made to their content, pedagogy, and use of 
technology with their students. They had to learn to use technology as the main part of their 
teaching instead of an add-on. They had to learn how to make technology work with their 
existing content and pedagogy. They had to change some content and pedagogy to adapt to using 
technology with their students. The teachers had to learn how to use Zoom or Google Meet to 
communicate with their students and adjust to not being face-to-face. They had to record 
themselves teaching. Many teachers who never used platforms like Google Classroom reported 
that they still use it because they now like it. Teachers used the Internet and different websites to 
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find activities and content instead of their teacher manuals and traditional curriculum. The 
teachers in this study noted they had to change how they were teaching because they could not 
hold small groups, do science experiments, or use other hands-on methods to teach. 

Most participants noted a substantial increase in workload and the need for self-reliance 
when developing ways to reach students during the school shutdowns in the spring of 2020. The 
teachers in this study reported not having many directions from their administrators because they 
were unsure of what to do. While many teachers received some guidance, the participants had to 
figure out what to do independently. The participants had to take on many new tasks such as 
contacting the parents and students, setting up their virtual classrooms, and learning the new 
platforms and apps. These tasks were not only unfamiliar but also time-consuming. Teachers 
spent many hours reaching out to parents, trying to find devices and connectivity for students. 
The teachers had to figure out how to use things like Zoom and Google Classroom. They had to 
spend the time to look for resources, websites, and materials that could be used online with their 
students. Many teachers delivered devices or materials themselves directly to the students’ 
homes. Many of the teachers in this study said they felt like they were on their own. Some said 
they had things being thrown at them without being told exactly how to do it. 

All participants expressed having experienced positive relationships with either 
colleagues or students during emergency remote teaching. The teachers in this study stated that 
the students looked forward to the Zooms or online meetings, even when the discussion was not 
academic. They felt it gave the students something to look forward to and created a sense of 
normalcy at a very uncertain time. It was not always academic, and sometimes the students and 
teachers would talk about what was going on in the world or show their pets. Some of the 
participants thought this made the students feel safe, and they enjoyed being able to check on the 
students, both physically and mentally. Many teachers felt that emergency remote teaching 
allowed them to get to know their students better, bond with them, and see certain qualities they 
may not have seen in a traditional format. All eight participants in this study noted positive 
relationships with their coworkers due to emergency remote teaching. The teachers worked 
together, sharing materials and ideas. They were texting and calling each other constantly. They 
found ways to get through this difficult time together, and they split up tasks to share the 
workload. Some teachers felt that emergency remote teaching brought them closer to their 
colleagues because they depended on each other. 


Implications for Future Practice 


School districts in the post-COVID-19 world must be ready for the possibility of a crisis- 
prompted teaching situation in the future. Changes can be made to education to prepare 
administrators, teachers, parents, and students to face something like a worldwide pandemic. The 
emergency remote teaching situation experienced by educators in the spring of 2020 offered 
many "lessons on preparedness that can and should be addressed in anticipation of another crisis 
teaching situation" (Pacheco, 2020). The research done in this study suggests school districts 
need to look at the measures they have in place for teaching during an emergency, such as a 
pandemic. 

First, the results of this study demonstrate the need for school districts to have a 1:1 
device program in place, where each student has a mobile device that they use in school and at 
home. Based on this study’s findings, educators felt unprepared to enter emergency remote 
teaching because their students did not have the devices necessary to connect with the teachers. 
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Second, school districts might also want to examine the professional development offered for 
technology use in the classroom and align it with teachers’ needs and daily use. School districts 
should consider ongoing professional development in the practical use of technology. The 
professional development has to be relevant and applicable to their classrooms. Third, teachers 
might want to increase their daily technology integration in their classrooms. A suggestion would 
be to get students and teachers familiar with the devices and use those devices in conjunction 
with a specific platform, such as Google Classroom, to communicate with students from a 
distance. Fourth, school districts should have a pandemic plan in place. A clear plan will help 
teachers face an emergency remote teaching situation with more confidence and less of a sudden 
increase in workload. The support and guidance on what is expected will save teachers time and 
make them feel less unprepared for an emergency. 


Recommendations for Future Studies 

There is value in studying teachers’ experiences with emergency remote teaching to 
evaluate what did and did not work and what we can learn to improve current and future practice 
(Hartshorne et al., 2020). The COVID-19 pandemic completely disrupted the education system 
and forced a wake-up call to strengthen public education (Kaden, 2020). This study was 
conducted to elementary classroom teachers’ experiences with emergency remote teaching 
during the COVID-19 pandemic. The data analyzed from this study suggests the need for further 
research on the phenomenon of emergency remote teaching. 

All school districts in the United States could benefit from additional research about 
teaching during the COVID-19 pandemic. It is recommended that this research study be 
replicated with teachers from various grade levels and subject areas. Future studies on 
emergency remote teaching should include teachers from middle school and high school. 
Studying middle school and high school teachers’ experiences and comparing them to that of an 
elementary school teacher would provide a basis for comparison to determine whether the 
themes for emergency remote teaching are seen at all grade levels. This comparative study would 
also highlight the potential differentiated needs of teachers transitioning to emergency remote 
teaching. 

A future study on emergency remote teaching could include the experiences of 
individuals other than the teachers. Studies could be done with administrators, parents, or 
students. The unique perspective of these groups of people could be compared and contrasted to 
teachers. The input from administrators, parents, and teachers could provide valuable 
information for educators regarding the best methods of conducting emergency remote teaching. 

Another recommendation for future studies is to conduct an identical study and include 
teachers from other states. Because every participant in this study had experience as a 
Pennsylvania teacher, it would be interesting to contrast the teachers’ experiences in other states. 
States experienced shutdowns due to COVID-19 at different rates and with different guidelines 
for emergency remote teaching. 

An additional recommendation for a future study is a study on existing pandemic plans. 
There is a need for future research studies to determine if school districts have adopted a 
pandemic plan or updated one previously in place. The details of these pandemic plans could 
provide valuable insight to school districts that lack one. School districts can use a study like this 
to provide support and specific guidance outlining what should be done for emergency remote 
teaching in their pandemic plan. 


158 


A study could be conducted comparing the experiences of male to female teachers. Since 
all of the participants in this study happened to be female, the experiences of male elementary 
teachers were not represented. 

Finally, a research study could compare schools with 1:1 device programs and 
educational platforms in place to schools that were not using technology daily with their students 
before the pandemic. The study could analyze the difficulty of the transition for these two 
groups. 


Conclusion 


The COVID-19 pandemic found educators across the world in a situation they never 
expected. The demands placed on teachers to suddenly switch to emergency remote teaching 
revealed that many school districts were unprepared to teach virtually due to a pandemic. This 
study aimed to examine the experiences of elementary classroom teachers who participated in 
emergency remote teaching. This study was necessary because it is important to step back and 
look at what teachers experienced during a worldwide pandemic and learn from those 
experiences. There was a dearth of research on online learning but minimal research on 
emergency remote teaching due to its timely nature. This study’s findings indicate that the 
experiences of elementary classroom teachers with emergency remote teaching were complex 
and characterized by emotional stress with the transition, technological and instructional barriers, 
changes in content, pedagogy, and technology, increased workload, and positive relationships. 
Future studies on emergency remote teaching should help to further the conversation about how 
to be best prepared for a crisis-prompted teaching situation in the future. 
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1. Introduction 
1.1 Education in Today's Japan 


The Ministry of Education, Culture, Sports, Science, and Technology (MEXT) (2020) 
has indicated the need to improve classes from the perspective of proactive, interactive, and 
authentic learning (so-called active learning) in the new curriculum guidelines to be implemented 
in high schools on an annual basis starting in 2022. It is essential to realize this improvement, not 
within a single lesson but a coherent unit or subject matter. In addition, since there will be no 
reduction in learning content from the existing curriculum guidelines, high school mathematics 
departments are required to plan lessons that consider teaching content and time allocation 
(MEXT, 2018). Thus, in today's Japan, students are expected to study more in-depth content in a 
limited amount of time. 

Therefore, flipped classroom teaching is an effective way to realize the improvement of 
teaching required today. 


1.2 Flipped Classroom 


According to Bergmann and Sams (2012), a flipped classroom is a learning method in 
which students do essential learning, such as the content of explanatory lectures, before class as 
homework, and learning necessary for retention of knowledge and cultivation of applied skills, 
such as tutorials and project learning. Abeysekera and Dawson (2015) found three critical 
components of flipped classroom teaching: 

1. Information-transmission teaching moves to outside of class time 

2. Lesson time used for learning activities 

3. Students require completing pre-and/or post-class activities to benefit from in-class 

activities fully 

These components suggest that it is possible to incorporate active learning into class time 
without reducing the learning content by introducing flipped classroom teaching. 
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One of the characteristics of flipped classroom teaching is the relationship between 
home-studying (preparation and review) and face-to-face teaching. According to Shinogaya 
(2012), all learning breaks down into three stages: prior learning, core learning, and post-learning. 
Shibukawa (2021) summarized the flow of learning in lecture-based and flipped classroom 
teaching based on three stages of learning (Table 1). 


Table 1. Difference between lecture-based class and flipped classroom 


Learning Stage Lecture-based Teaching Flipped Classroom Teaching 


Preparation Pre-Class Learning 
Ex) Reading Textbook Ex) Watching lecture videos 


Acquire new knowledge 
Soa ‘ Class Lesson 
Cee arene Ex) Knowledge Acquisition Class Lesson 


Ex) Engaging in developmental 
Revision activities 
Ex) Summarize Class Post-Class Learning 


Post Learning 


On the one hand, in lecture-based teaching, students learn the content of the class through 
preparation, acquire knowledge in the face-to-face class lesson, and practice problems and 
summarize the class content in the revision. On the other hand, in a flipped classroom teaching, 
the student acquires knowledge of the lesson's content in the pre-class learning, performs 
exercises in the face-to-face class lesson, and summarizes the content in the post-class learning. 
In this way, in the flipped classroom, students need to learn new knowledge on their own while 
studying at home. Therefore, the materials for the pre-class learning in the flipped classroom 
need to be designed so that the teacher's explanation of the study contents, the pre-class learning, 
and the face-to-face class lesson is considered one learning process. 

However, the implementation of prior learning can be problematic because there are no 
instructors present where learners can conduct prior learning. 


1.3 Problems with Pre-Class Learning in Flipped Classroom 


Some prior studies have shown that pre-class learning did not work well in flipped 
classroom teaching. One of the reasons is that learners do not carry out pre-class learning. For 
example, in Yamamoto et al. (2018) 's practice, a teacher needs to provide individual attention if 
students do not do the pre-class learning. Furthermore, Long et al. (2017) found that students 
might not prepare for pre-class learning, so teachers need to encourage their motivation for pre- 
class learning before in-class activities for all students to enjoy active learning. Wei et al. (2020) 
also stated that teachers need to encourage students to conduct pre-class learning to make in- 
class activities more active. Thus, the learner's implementation of prior learning is a necessary 
element for the success of the flipped classroom. 

Based on the above, we thought it necessary to examine what kind of support is needed to 
improve the rate of students' implementation of assignments. 
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2. Purpose 


It is still unclear what kind of support effectively encourages high school students to 
carry out their assignments. Therefore, this study aims to investigate the difference in the 
awareness of the students who submitted their assignments and those who did not submit their 
assignments, examine the factors that affect the students' implementation of the assignments, and 
propose support methods for each factor. 


3. Methods 

3.1 Participants of Research 

We surveyed 80 second-year high school students (40 students in each of two classes) 
enrolled in a public high school in Chiba Prefecture, Japan. The participants are taking a course 
called "Mathematics B," The unit covered in this study is "Space Vector." 
3.2 Flow of Research 

Figure | shows the flow of this survey. This survey was conducted within the "Space 
Vectors" unit in October 2020. The subject took a class on "Components of vectors and 
operations with components" within the unit. In the class, new knowledge was explained, and 
problems were practiced. In the review assignment, the students were required to practice five 


similar problems to the problems in the class. After the students submitted their assignments, a 
questionnaire survey was conducted on their motivation to perform the assignments. 


9th, October 2020 Class Lesson 
9th, October 2020 Introduction to Review Assignments 
12%, October 2020 FR @te)|(-esifeyam-lare me e-(el ale Mol MACS(¢ LA ta-val 


15%, October 2020 Questionnaire Survey 


Figure 1. Flow of Research 
3.3 Questionnaire survey 


A unique questionnaire was developed and administered to investigate the psychological 
situation and the learning environment in the students were performing the assignment. The 
questionnaire items were divided into three categories based on the different ways in which the 
students carried out the assignments: 

(1) When they carried out the assignments and finished them 

(2) When they carried out the assignments but did not finish them 

(3) When they did not carry out the assignments 
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The total number of questions was 17 items: (1) 7 items, (2) 5 items, and (3) 5 items, and 
each item used a 5-point Likert scale (1 = disagree, 5 = very much agree). In addition, they were 
asked about the most significant causes for each of the times when they performed the 
assignments and when they did not perform the assignments in an open-ended question. 


4. Results 
Among the 80 participants in this study, 53 (approximately 66%) who responded to the 


questionnaire without flaws were included in the analysis. Table 2 shows the implementation 
status of the assignments for the 53 participants in the analysis. 


Table 2. Status of students' implementation of the assignments 


Score (5-point scale) 


Group " Mu SD 
Submitted 42 4.36 0.84 
No-Submitted 11 - - 
total 53 


4.1. Reliability of the Questionnaire 


To investigate the reliability of the original questionnaire, Cronbach's alpha was 
calculated for the responses to the 17-item questionnaire. As a result, + = 0.73, the results of the 
questionnaire used in this study are considered reliable and will be analyzed. 


4.2. Attitudes toward Assignment Implementation and Environmental Differences 


In this research, a comparison of means was conducted on the responses to the 
questionnaires to examine whether there were differences in attitudes and environments between 
students who submitted the review assignment (submitted group) and those who did not submit 
the assignment (not submitted group). Shapiro-Wilk's normality test was conducted on the results 
of the questionnaire for each group, and no normality was found. In addition, there was a 
difference in the sample size of each group (submitted group=42, not submitted group=11). 
Therefore, Mann-Whitney's U test was adopted as the analytical method for comparing the 
means. 

We compared the differences in attitudes toward the implementation of the assignment 
between the submitted and un-submitted groups based on the responses to the first question 
(When they carried out the assignments and finished them) of the questionnaire survey. The 
results are shown in Table 3. 


Table 3. Differences in attitudes toward completing assignments 
Not 


(1) When they carried out the Submitted ; M.- 
assignments and finished them SSUES M2 i 


Mi SD1 M? SD2 


I do assignments because I believe r 
Hat haul aa ease ncn eae A2Q6 059° 3:59: 104° O71 136.50" 0.32 
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school. 


I do assignments because I believe 
2 that turning them in is related to 4.07 0.78 3.64 1.12 0.43 178.00 0.18 
my grades. 
I do assignments because it is 
necessary for me to do so to 
understand and master the course 
content. 
I do assignments because I know 
the purpose of doing them. 
I do assignments because I like 
5 learning about the subject 2.76 1.10 2.45 1.04 0.31 195.00 0.11 
assigned. 
I do assignments because I enjoy 
6 learning about the subject 2.71 1.15 245 1.21 0.26 204.00 0.08 


3.83 0.85 3.45 1.13 0.38 187.00 0.14 


3.67 0.85 3.09 1.14 0.58 148.507 0.26 


assigned. 
I do assignments well in advance 476 119 9.45 1.44 0.31 196.00 0.12 
of the deadline. 

n=53, 5-point Likert scale *p<.100 


As aresult of comparing students' attitudes toward "carrying out and completing 
assignments," there was a significant tendency toward the submitted group on the p<.100 
criterion for items (1)-1 and (1)-4. 

We compared the difference in the awareness of the submitting group and the not- 
submitting group regarding the case where they carry out the assignments but do not finish it, 
based on their responses to the second question (When they carried out the assignments but did 
not finish them) of the questionnaire survey. The results are shown in Table 4. 


Table 4. Differences in attitudes when assignments are done but not completed 


: ; Not 
(2) When they carried out the Submitted Saher ted M.- U ; 


assignments but did not finish them WW, SD; M>. SD? M2 


I do not finish assignments when I 
feel that they are too much. 
I do not finish assignments when I 
2 feel that I cannot solve the 2.50 1.09 3.27 1.35 -0.77 153.00° 0.24 
problems. 
I do not finish assignments when I 
3 do not know what to referto when 3.02 0.84 3.45 1.37 -0.43 162.00 0.22 


doing my assignments. 
I do not finish assignments when I 


4 do not have enough time due to 2.83 1.06 2.73 1.01 0.10 221.00 0.03 
assignments in other subjects. 
5 Ido not finish assignments whenI 395 1.31 3.00 1.34 0.05 225.00 0.02 


3.14 1.18 2.73 142 041 18450 0.14 
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don't have time for club activities, 
lessons, cram school, or other 
commitments. 


n=53, 5-point Likert scale *p<.100 


As aresult of comparing the students' attitudes toward "doing the assignment but not 
finishing it," there was a significant tendency toward the not-submitted group on the p<.100 
criterion for item (2)-2. 

We compared the differences in attitudes of the submitted group and the un-submitted 
group regarding the non-implementation of assignments based on their responses to the third 
question (When they did not carry out the assignments) of the questionnaire survey. The results 
are shown in Table 5. 


Table 5. Differences in attitudes when assignments are not completed 
Not 


(3) When they did not carry out the Submitted Submitted M1- U 
assignments a a oe ee 


Mi SD M> SD2 
I do not do assignments when I 49) 15g 973 1.10 0.17 217.00 0.04 


forget that I have them. 


I do not do assignments when they 3.48 


do not count toward my grade. 0.97 3.00 0.89 0.48 165.50 0.21 


I do not do assignments when I 
3 feel that they are not necessary for 3.64 1.03 3.00 0.77 0.64 143.00° 0.28 


my learning. 


I do not do assignments when I ; 
fecl that doing them is-ahassle. 3.83 0.88 3.18 0.98 0.65 150.50' 0.26 


I do not do assignments when I do 
5 not understand the purpose, 3.71 0.94 3.36 0.92 0.35 182.50 0.15 


meaning, or intent of doing them. 
n=53, 5-point Likert scale “p<.050, 'p<.100 


As aresult of comparing the students' attitudes toward not carrying out assignments, 
there was a significant difference in the submitted group based on p<.050 in item (3)-3. 
Furthermore, in item (3)-4, there was a significant trend toward the submission group at the 
p<.100 criterion. 


4.3. Categorization of Learner Characteristics of Assignment Implementation 


There was no significant difference between the submitted group and the un-submitted 
group when the difference in the attitude toward the assignments was examined by comparing 
the mean values. Therefore, we conducted a factor analysis to examine the factors that influence 
students' performance of the assignments. The questionnaire items were categorized into two 
types: items related to factors for performing the assignments and items related to factors for not 
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performing the assignments, and factor analysis was conducted on each of the two types of items 
to identify factors that influence the performance of the assignments. 

First, to examine the students' motivation to perform the assignments, factor extraction 
using the principal factor method and exploratory factor analysis using ProMax rotation were 
conducted on (1) of the questionnaire. The results are shown in Table 6. 


Table 6. Students' attitudes toward performing the assignment 
factor 


I I Il 
(1)-5 I do assignments because I like learning about the 0.94 0.02 0.01 


subject assigned. 
(1)-6 I do assignments because I enjoy learning about the 0.94 -0.03 0.03 


subject assigned. 
(1)-4 I do assignments because I know the purpose of 0.16 08 -0.02 


doing them. 
I do assignments because it is necessary for me to 


Questionnaire 


Clr “doisoito understand and itister the course. content maar SOE ane 
(1)-7 I do assignments well in advance of the deadline. 0.05 0.54 -0.18 
(1)-1 I do assignments because I believe that I should do O11 054 032 


the assignments at school. 
I do assignments because I believe that turning them 


Oe in is related to my grades. yee me 
Cronbach's alpha 0.94 0.69 - 
as 
factor correlation 0.16 


MW 0.03 0.31 
n=53, Kaiser-Mayer-Olkin's measure: .57, Bartlett's test: @=129.13. 
Factor extraction by principal factor method, ProMax rotation 


The results of the factor analysis showed that the motivation to perform the assignments 
was categorized into three factors. The factor I consisted of (1)-5 and (1)-6, Factor II consisted of 
(1)-4, (1)-3, (1)-1, and (1)-7, and Factor HI consisted of (1)-2. There was also a weak inter-factor 
correlation between factors II and HI. 

Next, to examine students' motivation to not perform the assignments, factor extraction 
using the principal factor method and exploratory factor analysis using ProMax rotation were 
conducted on (2) and (3) of the questionnaire. Also, all items were reversed because the 
questionnaire items were written in such a way as to ask for negative items. The results are 
shown in Table 7. 


Table 7. Students' attitudes toward not performing the assignment 
factor 
I I Il IV 


0.86 0.13 -0.06 -0.10 


Questionnaire 


I do not do assignments when I do not 


Oe understand the purpose, meaning, or intent of 
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doing them. (R) 


(3)-3 I do not do assignments when I feel that they 0.73. 0.08 0.06 -0.06 


are not necessary for my learning. (R) 
(3)-4 I do not do assignments when I feel that doing 0.50 -023 0.03 0.30 


them is a hassle. (R) ; 
I do not finish assignments when I don't have 
(2)-5 time for club activities, lessons, cram school, -0.04 1.01 -0.05 0.00 
or other commitments. (R) 
I do not finish assignments when I do not 
(2)-4 have enough time due to assignments in other 0.13 049 -0.01 -0.01 


subjects. (R) 
(2)-2 I do not finish assignments when I feel that I 0.15 005 085 0.00 


cannot solve the problems. (R) j : 
I do not finish assignments when I do not 
(2)-3 know what to refer to when doing my 0.15 -0.11 0.67 -0.17 


assignments. (R) 
(3)-1 I do not do assignments when I forget that I 0.10 0.00 -0.16 0.62 


have them. (R) 
(3)-2 I do not do assignments when they do not 0.08 -0.02 -0.06 0.58 


count toward my grade. (R) 
I do not finish assignments when I feel that 


(2)-1 theyare tao mice (R) 0.09 0.21 0.22 0.48 
Cronbach's alpha 0.73 0.67 0.63 0.59 
nie 
factor correlation oes 
I 0.16 0.35 
IV 037 0.41 0.49 


n=53, Kaiser-Mayer-Olkin's measure: .55, Bartlett's test: @=146.47. 
Factor extraction by principal factor method, ProMax rotation (R): Reverse items 


The results of the factor analysis showed that the motivation to perform the assignments 
was categorized into four factors. The factor I consisted of (3)-5, (3)-3, and (3)-4, Factor I 
consisted of (2)-4 and (2)-5, Factor II consisted of (2)-2 and (2)-3, and Factor IV consisted of 
(2)-1, (3)-1 and (3)-2. Also, weak inter-factor correlations were found between Factor I and 
Factor II, Factor I and Factor IV, and Factor II and Factor I, while moderate inter-factor 
correlations were found between Factor II and Factor IV, and Factor III and Factor IV, 
respectively. 


5. Discussion 
The above results suggest that the students in the submitted group feel more obligated to 
submit assignments than the students in the un-submitted group for the subjects of this study. In 


addition, it is thought that students are aware that they should perform and submit the 
assignments that are given to them regardless of the effect of performing the assignments on 
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their academic performance or the ease of performing the assignments. Therefore, it is possible 
that "submitting the assignment" itself is the purpose of the assignment. 

Furthermore, on the one hand, the results of the factor analysis for the questionnaire (1) 
suggest the following factors for the student's implementation of the assignment. 

e Students understand the purpose and need for the assignment 

e Students' interest in the subject matter is high 

e Students know that the assignment will be graded 

On the other hand, the factor analysis results for questionnaires (2) and (3) suggest the 
following factors that prevent students from performing the assignments. 

e Lack of time to complete the assignment 

e ©The assignment is too difficult for students 

These suggest that five factors affect students' performance of the assignments. The 
following five ways of supporting students to promote the implementation of their assignments 
can be identified from the above five factors. 

e Specify the purpose of doing assignments 
Conduct the class in a way that attracts interest 
Clarify the relationship to grades 
Allow sufficient time for submission / Give time estimates 
Adjust the difficulty level based on the student's grade 

Among these, "Specify the purpose of doing assignments" is particularly effective 
because it is consistent with the students' attitudes in the submission group, and it is easy for 
professors to provide support. 
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Abstract 


COVID-19 introduced a large percentage of the world’s students and teachers to virtual 
learning. The purpose of this study was to examine the changes in perception toward virtual 
medical education during the pandemic in the U.S. Twitter data (tweets) associated with the 
virtual educational experience in the medical fields was obtained. Three analysis methods (topic 
modeling, sentiment analysis, and longitudinal cluster analysis) were adopted for the study. 
Across the selected topics, we found differences in conversations in virtual medical education 
fields. The topical patterns by frequency, polarity, and subjectivity identified the needs of those 
involved in virtual medical education and the areas that need to be overcome to improve virtual 
learning. More findings and implications are further discussed. 

Keywords: virtual medical education, twitter, pandemic, COVID-19 


Introduction 


The COVID-19 pandemic has been changing our world significantly. In this situation, 
many fields mostly get a negative impact, but some do not. Virtual education can be the second 
case. Under the severe restriction of contacting others to prevent the spreading of the disease, 
virtual education grabs a chance to be utilized in most educational organizations. Even people 
who did not favor virtual learning and did not have experience of it face the situation where there 
is no choice but to receive virtual learning. In this situation, they might be able to find positive 
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aspects of it. On the contrary, the probability of finding virtual education's problems that have 
not been detected also increases by the increment of the frequency of virtual learning. Remote 
work and virtual education are likely to continue, albeit less intensely than at the pandemic’s 
peak. As the pandemic has brought the future to the present with communication, professional 
development, and technology in medical education, virtual learning may have influenced a shift 
in people’s perception of medical education during the pandemic. This leads to examining the 
changes in perception of virtual medical education to identify a need to improve the quality of 
virtual medical education. 


Literature Review 


The pandemic has had profound impacts on medical education globally. One of the fields 
that has the greatest resistance toward virtual learning is medical education. Many medical 
schools have adapted to virtual classes by altering their real-time clinical exposure to online 
modes (Rose, 2020; Ferrel & Ryan, 2020). Some schools echoed concerns over clinical 
experience and assessment during these times because practice-based learning is the backbone of 
medical education. Inevitably, most medical students’ clinical placements stopped, and learning 
in classrooms and laboratories was cancelled, leaving students to continue their studies remotely 
(Ahmed, Allaf, & Elghazaly, 2020; Ferrel & Ryan, 2020; Sahi, P. K., Mishra, D., & Singh, T., 
2020). Medical professionals also have cancelled their training to cope with pressures from cases 
of COVID-19, which is considered fundamental in their education, training, and progression 
(Gill, Whitehead, & Wondimagegn, 2020; Rajab, Gazal, & Alkattan, 2020; Sandars & Patel, 
2020). 

The emergence of social media platforms can be traced back to 1996, but truly emerged 
in their modern form in the early 2000s (Singh, 2019). With platforms such as LinkedIn, 
Facebook, Twitter, Instagram, YouTube, and more, the number of worldwide users has jumped 
from almost a billion in 2010 to 2.62 billion in 2018 (McFadden, 2018; Singh, 2019). Social 
media platforms allow users to share knowledge simultaneously (Toprak v.d., 2009, pp.28-84). 
In this vain, social media opens up new possibilities to understand people’s perception (Selwyn, 
2007). Studies show that Twitter, which is one of the most commonly used worldwide social 
network tools, is used all around the world as the most chosen educational tool (Elavasky, 
Mislan, & Elavsky, 2011; Feliz, Ricoy, & Feliz, 2015; Junco, Heiberger, & Loken, 2010; Park, 
2013; Rinoldo, Tapp, & Laverie, 2011; and Zainal & Deni, 2015). This research study was 
conducted to analyze the use of Twitter. 

The worldwide COVID-19 pandemic crippled health systems and closed schools across 
the globe. This challenge makes educational organizations, educators, and instructional designers 
fall into problematic situations. As we move forwards, a COVID-19 generation of students and 
doctors need to continue their education and training. While pandemics have historically created 
challenges, identifying these challenges is the first step in converting them into opportunities. 


Purpose of the Study 
The pandemic has provided an opportunity to investigate whether learners or education 
providers in medical education have positive or negative perceptions, or whether there is a 


change in their perceptions for virtual learning. These investigations would allow us to identify 
their needs in virtual medical education, the reason why they do not favor virtual learning can be 
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analyzed, and the way to break through the problems. This study, thus, aims to examine the 
changes in perception toward virtual medical education after COVID-19 pandemic in the U.S. 
using tweets. This study has the following research questions: (a) What are the topical patterns in 
tweets related to virtual medical education over the past five years?; and (b) What are the 
trajectories in the characteristics (i.e., frequency, polarity, and subjectivity) of the topical patterns 
on virtual medical education? 


Analytical Framework 


Given the nature of our research questions, the study adopted a quantitative research 
method to yield a comprehensive analysis. In this study, the change in perception toward virtual 
education of stakeholders in medical education organizations in the U.S. were examined by 
Twitter data (tweets). The topic modeling method is useful to find how many topics in the 
collected tweets dataset and what do the topics represent. Cross-validation is a type of model 
validation technique for assessing how the results of a statistical analysis generalize to an 
independent data set. Each topic can be allocated to individual tweets, and the change in the 
number of tweets in the topics for each year can be inspected to find the change in people's 
interest. Also, sentiment analysis is useful to find the changes in people's polarity and 
subjectivities in a period. The segment analysis allows to systematically identify, extract, 
quantify, and study affective states and subjective information. Lastly, longitudinal cluster 
analysis is useful to find groups that share trends among trajectories of the topics over a certain 
time. 


Method 


Data Collection 

In order to extract tweets that represent the perception toward virtual medical education, 
predefined keywords that can be categorized in four primary ways (virtual: 10, medical: 30, 
education: 24, and region: US) and Twitter Premium API were used. Two software tools, Python 
3.8 and Microsoft Visual Studio 2019, were used to execute this process iteratively for one year 
after the national emergency declaration for the COVID-19 pandemic (March 13th, 2020 - 
March 12nd 2021) and the period of four years precedent the declaration (March 13th, 2016 - 
March 12nd 2020). A total of 6,542 tweets over five years were collected. 


Preprocessing 

Before analysis, by following steps, the extracted dataset was cleaned to increase the 
clarity of the result. First, tweets that were too short thereby often meaningless (1.e., < 60 
characters) and were therefore excluded. Second, using the NLTK library (Bird et al., 2009) in 
Python, stop words such as pronouns, prepositions, and postpositions were filtered out. Finally, 
‘term frequency-inverse document frequency (tf-idf) was used to select words that frequently 
occur but are not shared across all tweets. 


Data Analysis 


Three primary analysis methods, the Latent Dirichlet Allocation (LDA; Blei, Ng, & 
Jordan, 2003), sentiment analysis, and longitudinal cluster analysis, were employed to examine 
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the changes in perception of stakeholders of medical fields toward virtual education after the 
COVID-19 pandemic situation. 

Latent Dirichlet allocation (LDA; Blei et al., 2003), so-called topic modeling, one of the 
unsupervised machine learning techniques, was employed to identify topics in a set of documents 
(e.g., tweets). The topic models (Griin & Hornik, 2011) package in R (R Core Team, 2020) and 
RStudio 1.3.1056 (RStudio Team, 2020) was utilized for LDA. As the initial stage of LDA, the 
optimal number of topics were determined (Zhao et al., 2015) using the Idatuning package 
(Nikita & Chaney, 2020). Figure 1 shows three different metrics of Griffiths (2004), Cao and 
Juan (2009), and Arun (2010). They all suggested 20—30 topics as optimally emerging in the 
tweets—i.e., with 20 - 30 topics, the Griffith value was maximized (approaching 1), and the Cao 
and Juan and Arun values were stabilized. 


Poh _| malice: 
0.00 e 


@ =Arun2010 
A CaoJuan2009 
@  Griffths2004 


45678910 15 20 30 50 75 


40 
number of topics 


Figure I. The optimal number of topics identified by different metrics 


Sentiment analysis was conducted for finding the authors’ emotional states using the 
TextBlob library in Python. Specifically, the tweets having a subjectivity value (range from 0 to 
1) greater than or equals to .50 were classified as subjective otherwise categorized as objective. 
The polarity and subjectivity were estimated for each tweet. Polarity has ranged from —1 to +1, 
and the tweets were classified by the polarity value— negative (polarity < —0.30), neutral (—0.30 
< polarity < 0.30), and positive (polarity > 0.30). 

The longitudinal cluster analysis, an unsupervised machine learning technique, could find 
groups that share trends (such as increasing, decreasing, steady) among trajectories of the topics 
over a certain time without human intervention. When there are too many objects to be 
compared, it is more effective and easier for interpreting that grouping objects to a smaller 
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number of groups according to characteristics (trends) and compare the groups rather than 
directly comparing each object; therefore, in this study, longitudinal cluster analysis was hired to 
find clusters of the topics that share unique joint-trajectories across the 5-years. The changes in 
three factors 1) frequency of the topics, 2) averages of polarities, and 3) average of subjectivities 
over the 5-year period were used to group topics, using kml (Genolini et al., 2013) package. 


Findings 


LDA found 30 topics from the extracted tweets, but only 24 were clearly interpretable 
and relevant to the study of those topics; therefore, the remaining 6 topics and their words were 
excluded from the analysis. The 24 selected topics were labeled as remote learning (topic 1), 
support (topic 2), virtual program (topic 3), family support (topic 4), nursing program (topic 5), 
social distance and safe (topic 6), Benefits (topic 7), anatomy class (topic 8), mental health (topic 
9), negative feeling (topic 10), professional development (topic 11), clinical study (topic 12), 
course assignment (topic 13), registration (topic 14), virtual meeting (topic 15), virtual medical 
lab (topic 16), quality of education (topic 17), stakeholders (topic 18), gratefulness of online 
learning (topic 19), working from home(topic 20), nursing classes (topic 21), hybrid modality 
(topic 22), virtual commencement (topic 23), medical training/certification (topic 24). 


From the sentimental and longitudinal cluster analyses, four clusters are identified with 
19 selected topics using their trajectories for the 5 years. The longitudinal clustering algorithm 
found three, four, and two unique trajectories of topics from the frequencies of the topic, 
polarities, and subjectivities across the 5-years, respectively (see Figure 2, 3, & 4). 
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Figure 2. Graphs for results of longitudinal cluster analysis by frequency 


176 


Calinski.Harabatz 
Sorted 


AW I SKB= 233.3%C- 2 292%0- 24.17% 


5 10 15 20 


Rerolling Times 


Figure 3. Graphs for results of longitudinal cluster analysis by polarity 
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Figure 4. Graphs for results of longitudinal cluster analysis by subjectivity 
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In frequency, cluster A (54.2%) showed the tendency of increasing frequency after the 
pandemic but compared to other clusters, the slope was gradual. As the graph shows, cluster B 
(33.3%) reported medium steeper increasing trend after the pandemic. The most dramatic 
trend—an exponential increase after the pandemic—was displayed in cluster C (12.5%). 

In polarity, the algorithm found four clusters. Cluser A (33.3%) shows mostly steady 
polarity around 0.1 after dropping in 2017, which means the topics belong to cluster A had been 
written in a neutral mood in the period. Cluster B (33.3%) and C (29,2%) displayed slight 
decrease tendencies after 2020, so it can be said that there were a few more negative tweets on 
the topics after the pandemic. However, cluster C has noticeably more positive tweets compared 
to cluster B during the whole period. Cluster D showed a very unique pattern that it was rapidly 


dropping in 2017, after then has been increasing regardless of the pandemic. Thus, it might be 
guessed that something probably happened in 2017 that influenced in the negative direction to 
cluster D which has only one topic, gratefulness about online learning (topic 19). However, after 
2017, the mood has been moving toward a positive direction. 


In subjectivity, only two clusters were suggested. As through the pandemic, cluster A 


(66.7%) showed a slight increase pattern that means people wrote more subjective tweets, and 
cluster B (33.3%) illustrated the opposite result (more objective viewpoint). All identified 
clusters by three factors and their topics are shown in Table 1. The cluster names were arbitrary 


assigned by the algorithm. For instance, Cluster A found by frequency is different from cluster A 


identified by polarity. 


Table 2. Identified clusters and their topics by frequency, polarity, and subjectivity. 


Cluster By frequency (Topic #) By polarity (Topic #) By subjectivity (Topic #) 
A professional development remote learning (1) remote learning (1) 
(11) benefits (7) support (2) 
clinical study (12) anatomy class (8) virtual program (3) 
course assignment (13) clinical study (12) nursing programs (5) 
registration (14) virtual medical lab (16) benefits (7) 
virtual meeting (15) quality of online course (17) anatomy class (8) 
virtual medical lab (16) working from home (20) mental health crisis service and 
quality of online course (17) medical training/certification support (9) 
stakeholders (18) (24) professional development (11) 
gratefulness about online clinical study (12) 
learning (19) course assignment (13) 
working from home (20) registration (14) 
hybrid modality (22) virtual meeting (15) 
virtual commencement (23) virtual medical lab (16) 
medical training/certification working from home (20) 
(24) virtual commencement (23) 
medical training/certification 
(24) 
B support (2) virtual program (3) family support (4) 
virtual program (3) social distance and safe (6) social distance and safe (6) 
family support (4) mental health crisis service negative feeling (10) 
nursing program (5) and support (9) quality of online course (17) 
benefits (7) negative feeling (10) stakeholders (18) 
anatomy class (8) course assignment (13) gratefulness about online 
stakeholders (18) learning (19) 
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negative feeling (10) nursing classes (21) nursing classes (21) 


nursing classes (21) hybrid modality (22) hybrid modality (22) 
C remote learning (1) support (2) - 

social distancing and safe (6) family support (4) 

mental health crisis service nursing programs (5) 

and support (9) Professional development 


(1) 
registration (14) 
virtual meeting (15) 
virtual commencement (23) 
D - gratefulness about online - 
learning (19) 


Discussion and Implications 


The study examined the changes in perception of medical fields toward virtual education 
on tweets for the last five years including the COVID-19 pandemic period. Across 24 selected 
topics, major themes were identified in conversations in virtual medical education fields. First, 
medical education schools were unprepared and ill-equipped to handle the overwhelming 
obstacles and immediate needs (e.g., new planning, support, accessibility, mental health crisis, 
registration). Second, COVID-19 has accelerated reshaping of medical education (e.g., 
leadership, safe learning environment, clinical study, group activities, virtual lab, quality of 
online course, hybrid). Third, virtual medical education was not only challenging for the last 12 
months but they also look to the future possibilities (e.g., benefits, professional development, 
experiential learning). 

The results from the sentimental and longitudinal cluster analyses identified clusters and 
their topics by frequency, polarity, and subjectivity. The trajectories information over the past 
five years helps explain a need in virtual medical education and the reason whether they favor 
virtual learning or not. Virtual education has grown rapidly since the pandemic. Medical 
education has also seen a rapid increase over the past year. As formal medical education had to 
move to remote format for social distancing, mental health crisis also increased (Cluster C). In 
order to reduce these side effects, many supports have also been increasing (Cluster B). 

The polarity of the tweets over the period shows that there were more positive statements, 
especially since the pandemic of gratefulness for virtual education has seen the highest rise in 
positivity on average. Conversely, there was a slight decline in other clusters (Clusters A, B, & 
C, indicating many difficulties and challenges for virtual medical education. 

Lastly, the subjectivity. Subjective statements about virtual medical education generally 
represent personal opinions, feelings, or judgments, while objective ones represent factual 
information. Both clusters identified were found to be subjective due to the nature of expressions 
on social media. In the past, there was a large difference between Cluster A and B, but recently 
the gap has been gradually narrowing. Social safety, negative emotions, and family support have 
been expressed a more subjective expression on their tweets. On the other hand, as for the 
general views of virtual medical education, the effort was reflected by increasing objectivity 
during the period (Cluster A). 

Virtual education continues, although less intense than at the peak of the pandemic. We 
hope that this study contributes to broadening our understanding of virtual medical education. By 
understanding their needs for virtual medical education, we expect to be connected in a concrete 
way to how to solve problems in virtual medical education. 
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Abstract 


Students construct knowledge during asynchronous online discussions (AODs). 
Discussion postings show students’ cognitive presence (CP) patterns and content learning levels. 
The purpose of this longitudinal field observation study was to investigate the patterns of, and 
relationships among, the same cohort of graduate students’ cognitive presence and content 
knowledge level demonstrated within and across the AODs in two consecutive courses over two 
semesters. Content analysis of online discussion postings, descriptive and correlational statistics 
were used to analyze data. The results indicated that the exploration was the most active phase 
that emerged in AODs, followed by the integration, triggering event, and resolution. There was a 
significant change in the integration phase of CP and content knowledge levels over time. 
Moreover, the results yielded significant relationships among CP patterns and content knowledge 
levels. The findings have important implications theoretically in confirming the CP patterns that 
emerged in AODs and practically by identifying the dynamics of each of the CP phases and their 
associations with content learning. 


Keywords: Asynchronous Online Discussions; Cognitive Presence; Content Knowledge 
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Introduction 


For decades, asynchronous online discussions (AODs) have been used as a social, 
collaborative learning environment in higher online education. Social constructivism views 
learning in such an environment as an act of constructing knowledge through socially informed 
cognitive engagement where participants share, debate, listen, reflect, and confirm their 
knowledge in the context of communicating with others (Jonassen et al., 1995; Johnson & 
Johnson, 1996; Vygotsky, 1978). 

AODs are primarily designed to prompt learners to engage cognitively to enhance their 
content learning through social interactions. Current research in AODs often studies cognitive 
engagement using the cognitive presence (CP) construct of the Community of Inquiry (Col) 
framework to examine students’ learning process (Garrison et al., 2000). CP defines ways that 
students mentally navigate the social learning process; how they consider new problems, seek to 
develop understanding, and share understanding with their learning community during the 
process of validating and modifying their understanding (Akyol & Garrison, 2011). 

In AODs, students co-construct their knowledge through iterative interactions and 
collaborations with others in their learning community (Hew & Cheung, 2003; Joksimovic et al., 
2014). Such collaborative interactions can prompt co-regulation and metacognition among those 
participating in communication events, activating, and enhancing cognitive processing within the 
individual and across the community (Jonassen et al., 1995). 

AODs can activate a student’s cognitive processing beyond surface learning into deeper 
learning through CP (Garrison et al., 2001). “In AODs, learners who have different ideas and 
prior experiences share their thoughts, read, reflect on, and react to others’ postings using their 
cognitive processes (behavior) to collaboratively create a richer understanding of the content 
topic at hand (learning)” (Koszalka et al., 2021, p, 3). Such interactions represent higher-order 
thinking (Anderson et al., 2001) and suggest behaviors that indicate the construction of deeper 
content knowledge (Akyol & Garrison, 2011; Garrison et al., 2001; Jonassen et al., 1995). 

However, studies have shown that learning expectations for multiple CP phases and 
deeper content learning in AODs are not always achieved (Chen et al., 2019; Koszalka et al., 
2021; Koszalka et al., 2019). Researchers have examined factors that may contribute to 
exploration activities during cognitive engagement processes to understand these deficiencies. 
But few studies were found that directly studied the relationships among cognitive presence (CP) 
and level of content knowledge demonstrated in AODs. Koszalka et al. (2021) recently 
suggested a significant relationship among levels of idea exchange, content focus, and 
surface/deep learning in AODs, but only studied this phenomenon in several one-time, short 
AOD sessions, not directly investigating CP. Since individual cognitive engagement can result in 
different cognitive development outcomes, e.g., levels of demonstrated content learning 
(Jonassen et al., 1995; Koszalka et al., 2021), it is important to unpack the relationships between 
CP and content learning levels. And how learners enter deep content learning during AODs with 
the development of CP processes and whether they maintain or build different patterns and levels 
of deep learning in subsequent AOD experiences are lack of study. Therefore, it is necessary for 
us to explore how CP presences and content learning changes over time. Such studies may 
inform design strategies for conducting effective AODs. 

The current study focused on studying the relationships among each phase of CP (trigger, 
exploration, integration, resolution) and the level of content learning demonstrated within the 
same cohort of students, across two consecutive core courses, offered over two semesters 
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(longitudinal study). We hope to identify whether students who engage in various surface and 
deep learning strategies during AODs in an early course benefit from this initial experience and 
continue to engage in deeper learner behaviors in subsequent AODs. These questions suggest 
looking at longitudinal patterns of cognitive presence and learning levels across multiple, 
sequential AODs within and across courses. Therefore, the present study aims to add our 
understanding of the relationships between CP and depth of content learning and how these 
patterns form (or not) longitudinally. 


Research Questions 


The current study involved analyzing the AODs transcripts. We collected the convenient 
data from the same cohort of 12 graduate students from two courses. These students were 
pursuing a master’s degree in instructional design. Before taking the two courses, all students 
have had experience in online courses. IRB exempt status was granted for this study. Female and 
male students were 2 (16.7%) and 10 (83.3%), respectively. The average age of the cohort is 44. 
All participants have military experience. The average length of their military service is 22 years. 
Two of them are veterans, and the rest of them are active-duty service members. 

Students’ CP patterns were coded and examined using Garrison et al.’s (2000) Col 
framework. Content knowledge levels were identified and analyzed using Anderson et al.’s 
(2001) updated version of Bloom’s taxonomy and previously developed protocols from Koszalka 
et al. (2021). There were three research questions: 

1. What patterns of cognitive presence and content knowledge levels emerge within and 
across the AODs? 

2. How do the cognitive presence patterns and content knowledge levels of the same cohort 
of students change (or not) over time in the two courses investigated? 

3. Is there any significant relationship between cognitive presence patterns and content 
knowledge levels of the same cohort of students across multiple AODs? 


Method 


The transcripts of 276 postings from Course A and 273 postings from Course B were 
retrieved from the Blackboard LMS, downloaded, cleaned, and saved in the qualitative data 
analysis software MAXQDA before coding. There was no information related to students’ 
identities. The cohort of students participated in the AOD of Course A in the fall semester and 
then participated in the AOD of Course B in the spring semester of the same academic year. 

We applied a quantitative content analysis approach, transferring students’ postings into 
quantifiable codes. We followed the steps of a content analysis suggested by Neuendorf (2017, 
P. 67-69): 

Step 1. We collected data that fit our research purpose 

Step 2. We selected a single meaning as the unit of analysis. 

Step 3. The two coders in our team independently divided each posting into one meaning 
and then dismissed the inconsistencies through a discussion. 

Step 4. The two coders coded the meanings with two separate existing coding 
frameworks. The first is cognitive presence from Garrison et al. (2001), and the second is content 
level from Koszalka et al. (2021). The first four categories of the coding framework focused on 
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the cognitive engagement process in the course content. The last two categories focused on the 
level of content learning (surface or deep) demonstrated in students’ postings. 

Step 5. The two coders coded the first 200 meanings of the AODs for checking the inter- 
rater reliability. 

Step 6. The two coders discussed the inconsistencies, re-coded the 200 meanings to 
improve inter-rater reliability, and finished coding the rest of the meanings in the AODs. 

Finally, relying on the numerical data in terms of the codes, we run descriptive and 
correlational statistics through the SPSS software for finding the patterns (RQ1), changes (RQ2), 
and relationships (RQ3). 


Results 


RQ1. What patterns of cognitive presence and content level emerge within and across the 
AODs? 

Descriptive statistics were generated to understand the patterns. Through the content 
analysis, we produced 1630 meanings (n=1630) hidden in the postings of the AODs. 

While all the four phases of cognitive presence appeared in the meanings, the ratios of 
their frequencies were different. The exploration phase was identified from 1382 meanings out of 
1630, having the highest ratio (84.8%) among the four phrases. The integration phrase took 
second place with 349 meanings. Only 152 and 51 meanings correspondingly include the 
triggering event phrase and the resolution phrase. 

The proportions of the content levels are only slightly different compared to the 
proportions of CP. 947 meanings were coded as content low, while 664 meanings were coded as 
content high. Only 19 meanings existed for social purposes and did not have any content relating 
to the discussion topics. 


RQ2. How do the cognitive presence patterns and content knowledge levels of the same 
cohort of students change (or not) over time in the two courses investigated? 

To better understand whether and how the cohort’s patterns change over time in the two 
courses, we also coded each meaning of the AODs with the posted week as a time label. The 
AOD in Course A lasted five weeks, and the AOD in Course B lasted two weeks. The number of 
generated meanings from Week | to Week 7 are 197, 167, 190, 189, 180, 247, and 460. These 
numbers clearly demonstrated a tendency for the same cohort of students to become more 
engaged in the last two weeks as they created more meanings. 

Figure 1 shows the changes in the CP and content level in the AODs over time. When we 
observed the changes from a seven-week range, both the cognitive presence and content level 
fluctuated over time and thus did not show a consistent tendency. With low mean values, both 
the triggering event and resolution phrases are stated at low mean values and fluctuate less than 
other phrases. This indicated that students rarely played as an ice breaker (1.e., the role triggering 
a new topic) and problem-solver (i.e., the role posing a solution to a problem) in any time spot of 
their AODs. Comparatively, the mean value of the exploration phrase stayed high all the time, 
implying that students were more likely to play the explorer (i.e., the role providing pieces of 
content) and stick to this role during the whole AOD process. The tendency of the integration 
phrase fluctuated less in the first five weeks, but it showed a sudden increase in the last two 
weeks. This could suggest that after finishing the AODs of Course A, the cohort of the students 
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promoted their ability to integrate pieces of information into a whole new meaning unit as they 
were more willing to play the role of synthesizer. 

The tendencies of content level fluctuated more fiercely compared to the cognitive 
presence tendencies. We were surprised that the two tendency lines of the content low and 
content high were highly symmetrical and even looked like mirror effects. It implied a strong 
correlation effect between the content low and the content high. In the weeks of Course A, the 
content low tendency was leveling up in the first three weeks, peaked in the fourth week, and 
then fell sharply in the last week. Oppositely, the content high tendency was leveling down in the 
first weeks, reached a valley in the fourth week, and then went back to the situation close to the 
start point. 

Figure 1. AOD Patterns Over Time 


Course A H Course B 


=== TriggeringEvent 
= Exploration 

=== Integration 

=== Resolution 


==ContentLow 
== ContentHigh 


Pattern Mean 


Time (Week) 


RQ3. Is there any significant relationship between cognitive presence patterns and 
content knowledge levels of the same cohort of students across multiple AODs? 

Inferential statistics were generated for understanding the relationships among the 
cognitive presence phrases and content knowledge levels. The matrix below (Table 1) shows the 
significant correlations. The content high variable was strongly and significantly correlated with 
the content low variable negatively (r=-.836, p<0.01). It could indicate that students rarely 
invested their endeavors in creating meaning with both content levels in the AODs. Another 
strong and significant correlation occurred between the exploration and integration phase 
variables (r=-.674, p<0.01). It revealed that the exploration phrase and integration phrase hardly 
co-existed in a single meaning of the AODs. In other words, students tended to stay with either a 
deductive phase (i.e., exploration) or an inductive phase (i.e., integration) rather than involve 
both when they created an AOD meaning. 

Some correlations were found comparatively weak but significant. It seems that the 
resolution phase of CP was slightly correlated with the content low variable negatively (r=-0.09, 
p<0.01), but it was slightly correlated with the content high variable positively (r=0.081, 
p<0.01). It suggested that in the resolution phase, students preferred to stay at the content high 
level. The exploration phase variable was positively correlated with the content high variable 
(r=0.08, p<0.01). It suggested that when students explore information for building the meanings 
of the AODs, they tend to use the content with their higher levels of thinking (e.g., analogy, 
comparison). 
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Regarding the time variable, the data (r=-0.056, p<0.05) indicated that content level 
might increase as time went on. Furthermore, students seemed to become more cognitively 
engaged in the integration phase (r=0.083, p<0.01) and less cognitively engaged in the resolution 
phase (r=-0.063, p<0.05) when the AODs were closer to the end. 


Table 1. Correlation Matrix for All the Patterns 


ContentLow ContentHigh TnggenngEvent Integration Exploration Resolution Time 

ContentLow _ Peatson Comelation | ara -.034 -.021 =.007 

Sig, (2-tailed) 000 166 394 786 <.001 025 

N 1630 1630 1630 1630 1630 1630 1630 
ContentHigh Pearson Conelation 836" 1 005 012 (cco oa ay -.018 

Sig. (2-tailed) 000 851 638 001 001 456 

N 1630 1630 1630 1630 1630 1630 1630 
TriggeringEvent Pearson Conelation -.034 005 1 -.044 -.046 -.046 -.040 

Sig. (2-tailed) 166 851 076 062 066 105 

N 1630 1630 1630 1630 1630 1630 1630 
Integration Peatgon Conelation -021 012 -.044 1 ey. 

Sig. (2-tailed) 394 638 076 <.001 506 <.001 

N 1630 1630 1630 1630 1630 1630 1630 
Exploration Pearson Comtelation -007 080"* =046 | 674" 1 eee -.035 

Sig, (2-tailed) 786 001 062 <.001 004 153 

N 1630 1630 1630 1630 1630 1630 1630 
Resolution Pearson Conelation -.090"" os1*" -.046 -.016 Pore 1 

Sig. (2-tailed) <.001 001 066 506 004 010 

N 1630 1630 1630 1630 1630 1630 1630 
Time Pearson Conelation -.056" -.018 -.040 083"" -.035 -.063" 1 

Sig. (2-tailed) 025 456 105 <.001 153 010 

N 1630 1630 1630 1630 1630 1630 1630 


** Correlation is significant at the 0.01 level (2-tailed) 
* Correlation is significant at the 0.05 level (2-tailed) 


Discussion 


Our analysis of the first research question suggested that the exploration phase was the 
most active in AODs, followed by the integration, triggering event, and resolution. The highest 
ratio of exploration frequencies and the lowest ratio of resolution frequencies in AODs is 
consistent with previous research (Chen et al., 2019; Garrison et al., 2001; Meyer, 2003; 
Vaughan & Garrison, 2005). Some previous studies pointed out that it was difficult for students 
to reach the integration phase (Akyol & Garrison, 2008; Garrison, 2007; Kanuka et al., 2007; 
Vaughan & Garrison, 2005), but in this study, the integration phase is the second most active 
phase emerged in AODs. This finding aligns with Akyol and Garrison's (2011) study, revealing 
that the integration phase is active (Akyol & Garrison, 2011). The different results on cognitive 
presence patterns that emerged in AODs might be explained by different instructional designs of 
courses, including types of prompt questions (Chen et al., 2019; Garrison, 2007) and duration of 
AODs. The prompt questions for causes seeking and problem-solving are more likely to facilitate 
students to connect and synthesize ideas to construct their meanings and provide solutions, that 
is, to reach the integration and resolution phase of cognitive presence. The length of AODs also 
can affect students to achieve the integration and resolution phase. The longer students 
participated in AODs, the higher the possibility of reaching the integration and resolution phase. 
As for the content knowledge levels, the previous study, which explored the evidence of student 
content learning levels in a single course, showed that students' content-low postings were 
dominant in AODs (Koszalka et al., 2021), which is consistent with our study. 
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Our analysis of the second research question suggested that both cognitive presence 
patterns and content knowledge levels could change over time. Among the cognitive presence 
patterns, integration and resolution phases fluctuate a lot more than the other phases. Both the 
triggering event and exploration were at the highest levels at the beginning of the course weeks, 
and then there was a decreasing trend, finally going back to a higher level during the late weeks. 
Both the integration and resolution reached the highest levels during the mid and late course 
weeks and then presented a decreasing trend. These findings highly align with Sezgin's (2020) 
study. However, Sezgin (2020) conducted their study only in a single seven-week online course. 
Since this longitudinal study focused on changes over time in two courses across two semesters, 
something unique we found about the changes was that integration was the only phase of 
cognitive presence increased in the second course, which might be explained by the type of 
discussion questions and time. Students are more likely to reach the integration phase with more 
structured questions, and they can better integrate ideas from different sources with practice and 
efforts through AODs (Akyol & Garrison, 2008; Sezgin, 2020). For the content knowledge 
levels, we found that there were more content-low postings during the early weeks in which 
students tended to focus on sharing facts, a summary of reading materials, and their general 
understanding of the discussion topics. As time goes by, students gradually connect their 
understanding of the discussion topics with their personal experiences to demonstrate higher- 
order thinking skills. 

Our analysis of the third research question suggested the correlations between cognitive 
presence patterns and content knowledge levels. Such correlations may enlighten the 
instructional design of AOD in multiple aspects. First, the negative correlation between the 
content low variable and content high variable indicates that students tend to stay at a single 
content level (i.e., low or high) when constructing an independent meaning in the AOD. The 
correlation between the exploration and integration phases demonstrates a similar pattern: 
students focus on only one cognitive presence phrase when they construct meaning. Therefore, 
the instructors may need to require or encourage students to construct multiple meanings in a 
posting so that multiple content levels and phrases of cognitive presence can occur in the AOD. 
A tested instructional approach to making this happen is to design structured prompts for 
eliciting diverse meanings in a single posting (Darabi et al., 2013). Structured prompts often 
include sequential questions requiring students’ responses in different cognitive or content levels 
(Darabi et al., 2011). A group of researchers has found that structured prompts would lead to the 
robustness of meanings with various cognitive learning indicators in the AODs (Darabi et al, 
2011; DeNoyelles et al., 2014; Lee 2012). Second, both the exploration phrase and resolution 
phrase were significantly correlated with the content high variable. It suggests that if students are 
expected to engage in the AODs cognitively, they may need to equip their high-level content 
learning skills, such as comparing different concepts and creating real examples. The study 
conducted by Koszalka et al. (2021) involved a series of pre-work activities prior to the AODs. 
In the pre-work activities, students had the chance to read the assigned content and practice their 
high-level content learning skills by reflecting on some scaffolded questions. The result proved 
that students were more cognitively engaged in the AOD after taking the pre-work activities. 
Thus, preparing students for some high-level content learning skills in advance can benefit 
students' cognitive presence in the AOD. 
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Conclusion 


The research findings indicate that the exploration was the most active phase in AODs, 
followed by the integration, triggering event, and resolution. There was a significant change in 
the integration phase of CP and content knowledge levels over time. Moreover, the results 
yielded significant relationships among CP patterns and content knowledge levels. To encourage 
multiple CP phases and achieve deeper content levels in AODs, the instructors may need to 
require students to construct multiple meanings in a posting. For example, provide specific 
participation guidelines, design a timely feedback system, use structured prompts to elicit diverse 
meanings, or prepare students some high-level content learning skills in advance through well- 
designed pre-work activities. Future research may include but is not limited to the following list 
of potential empirical research: 1) Collect discussion transcripts in a longer time range (e.g., five 
years) to examine students' CP patterns and content learning levels demonstrated in AODs; 2) 
Compare the CP patterns and content learning levels of military students with non-military 
students to examine if students' characteristics influence their performance; 3) Integrate self- 
reported surveys and interviews to triangulate the findings of content analysis. 
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The Historical Development of Deep Learning and Its Research Trend: 
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Abstract: Deep learning has recently received unprecedented attention from governments, schools, 
social institutions, and the media. The puspose of this study is to present an in-depth understanding 
of relevant research published in top-tier journals from 1976 to 2019 through a systematic review 
method. 

The results reveavled: (1) four phases of deep learning research: dormant period, germination 
period, emerging period and rapid period; (2) the deep learning research is concerned more about 
process, little discussion on flexibility; (3) the concept shifted in two directions, one is from the 
pursuit of understanding to the pursuit of transfer and the other is from the focus on process to the 
focus on outcomes. Based on the findings, this study proposed a landscape to the concept of deep 
learning to deliever a full understanding of the deep learning and we suggest more research should 
focus on the problem of deep flexibility with the empowerment of smart classrooms. 

Keywords: concepts evolvement; deep learning; learning architecture; literature review; research 
trend; smart classroom 
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Influenced by the concept and framework of core literacy education, the deep learning 
movement has emerged. In addition, a series of education and teaching reforms triggered by 
information technology, especially AI technology, deep learning has once regained the wide 
attention of the academic community and the general public, especially at the time when machine 
defeated Lee Sedol, the world champion of Go. Deep learning in education has received 
unprecedented attention and recognition from governments, schools, social institutions, and the 
media (Zhu, & Peng, 2017). Currently, deep learning now is extremely valued among the school 
leaders (Johnson et al., 2014), and the shift to deep learning has also become a long-term trend 
driving the application of educational technology (Freeman, Adams , &Cummins, 2017). How to 
promote students' deep learning and cultivate their deep learning abilities has been an increasing 
important topic for an education reform. 


Deep learning is not a newly emerging concept. Since 1976 when it was first time proposed, deep 
learning has experienced more than 40 years ‘ development and evolution. While there is little 
discussion regarding how deep learning developed, how the concept evolved, what are the current 
research bottlenecks we might need to focus on and what are the potential solutions, this study is 
to explore those questions and hope to help scholars and teaching practitioners construct a 
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comprehensive understanding of deep learning and accurately grasp the orientation of deep 
learning research and development. By using a systamacti review method, this study deeply 
reviewed articles indexed in SSCI ranging from 1976 to 2019. 


1.Literature Sources and Analysis Methods 


According to the literature dispersion theory of most key literature is usually concentrated in 
a few core journals, this study uses the paper title names of "deep(er) learning, deep approach(es), 
deep strategy/strategies" to search articles indexed in SSCI on the Web of Science database. The 
obtained documents are selected according to the following selection criteria (Petticrew, &Roberts, 
2006): a) not a duplicate article, b) the topic should be about education and teaching, c) the 
publication date is between 1976 and 2019. A total of 109 arctiles were included for the further 
analysis. 

First, the trend of deep learning research popularity was explored using econometric analysis 
and visualized through social network centrality analysis. Content analysis was used for an in- 
depth understanding of the research bottleneck and possible solutions of deep learning. The social 
network centrality analysis was carried out according to the following steps: a) data cleaning (e.g., 
term unification); b) importing keywords of the literature into BICOMB2 software for a word 
frequency analysis; c) generating co-occurrence matrix through co-occurrence analysis of high- 
frequency keywords with word frequency greater than or equal to 3; d) importing co-occurrence 
matrix into UCINET6 software for the social network centrality analysis using the NetDraw tool. 


2.The trend and status of deep learning 


Content analysis results show that international scholars generally believe that the concept of 
deep learning was first proposed by Maton and Saljé6 in 1976, even though they did not use the 
term, deep learning. In fact, this term was gradually adopted after the 1990s, and the earliest user 
was the scholar Valerie Malhotra Bentz (1992). 

2.1. The trend of deep learning research 

Econometric analysis shows (see Figure 1) that deep learning has not been widely concerned 
by scholars until 21st century. Throughout the history of deep learning research, the trend of deep 
learning research has gone through four periods: dormant period, germination period, emerging 
period and rapid period. 
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Figure 1 The annual Distribution of Deep Learning Literature 


2.1.1.Dormant Period (1976-1995) 

Since the birth of deep learning in 1976, deep learning has entered a long dormant period (see 
Figure 1). Although the literature of this period is not available in econometric statistics, we found 
there were still a group of scholars dedicated to deep learning research. John Biggs, professor of 
educational psychology, is one of the representatives. After in-depth research, they published many 
interesting results. The two most famous results are the SPQ Learning Process Scale and the SOLO 
taxonomy. The first one is to measure deep/superficial learning strategies and motivations. (Biggs, 
1978), the second one is for measuring the results of deep learning (Biggs, &Collis, 1982). These 
two outcomes are still commonly used tools for scholars to measure the process and results of deep 
learning. In fact, after the 1970s, computers have gradually replaced some conventional tasks and 
manual tasks, and began to assist people in completing some unconventional tasks (Murane , 
&Levy, 1996), and promoted changes in in the enterprise's demand for talent structure: The 
demand for conventional skills has dropped sharply, and the demand for skills such as 
communication skills and expert thinking has soared (Conley, &Darling-Hammond, 2013). 
However, content analysis shows that deep learning at this time focuses more on the opposite of 
shallow learning (conventional memory, lack of deeply understanding, and just coping with 
exams), and it has not been in line with the above-mentioned social situation. This may be the 
reason why deep learning did not attract widespread attention and entered a long dormant period 
at this phase. 
2.1.2.Germination Period (1995-2007) 

Literature statistics show that in 1995, Hoon and others (1995) from Nanyang Technological 
University in Singapore applied deep learning strategies to high school chemistry classes for the 
first time to encourage students to visualize abstract concepts and explore the connections between 
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numerous chemical facts. Since then, deep learning research has entered a germination period, 
during which a small number of scholars’ research and application results have appeared one after 
another. Content analysis shows that the concept of deep learning during this period was similar 
to the dormant period, still focusing on process. But in addition to exploring more ways/strategies 
that can promote deep learning, scholars have begun to explore formative evaluation and the 
support of various learning environments. 

2.1.3.Emerging Period (2007-2016) 

After entering the 21st century, various international organizations and countries around the 
world have devoted themselves to exploring what kind of new century talents should be cultivated. 
As a result, a variety of talent competency frameworks emerged. For example, the OECD released 
the DeSeCo competency framework in 2003, the European Union released the key competences 
for lifelong learning in 2006, and the P21 released the 21st century learning framework in 2007. 
Those frameworks had prompted the research of deep learning to enter the emerging stage. During 
this period, influenced by these key competency frameworks, an international wave of deep 
learning movement emerged: In 2010, the Hewlett Foundation of the United States initiated a 15- 
year deeper learning strategic plan (The William and Flora Hewlett Foundation, 2012). Asia 
Society and other ten institutions have also worked together to promote the spread of deep learning 
experimental schools throughout the United States (Alliance for Excellent Education, 2017). 
Moreover, the Victoria University in Canada launched a global deep learning initiative, 
cooperating with more than 1,000 schools in 10 countries to seek solutions for deep learning 
changes (NPDL, 2017). In 2015, the United States even issued a report to take deep learning as 
the national policy of education in the 21st century (National Association of State Boards of 
Education, 2015). 
2.1.4.Rapid Period (2016 — present) 

In addition to the fact that the United States regards deep learning as a national education 
policy, the event signaling the arrival of the rapid development period of deep learning is when the 
intelligent robot AlphaGo defeated the world go champion Lee Sedol in 2016. This event once 
increased the concern that machines would replace humans, and prompted scholars to reflect on 
questions such as "since humans can teach machines to learn deeply, why can't we teach children 
to learn deeply in school?". During this period, deep learning research has been improved in all 
aspects: from concept expansion, strategies testing, model construction, to mechanism exploration, 
evaluation development, and then to subject application. During this period, deep learning enabled 
by technology, especially intelligent technology, has become an emerging research trend. In 
particular, the release of the Core Competences for Chinese Student’s Development in 2016 added 
a powerful fuel for deep learning research. Given this situation, Chinese Scholar Kekang He (2018) 
asserted that deep learning has regained the widespread attention of academia and even the general 
public (according to the data, the first attention occurred during the Rapid Period). 
2.2.The status of deep learning research 

Figure 2 shows the research status of deep learning formed by the analysis of social network 
centrality. The larger the box, the higher the centrality, and the distance between the boxes reflects 
the close relationship between keywords. 
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Figure 2 Community diagram of high-frequency keywords in deep learning literature 

On the whole, the research of deep learning is biased towards promoting students' deep 
participation in learning and adopting advanced learning strategies. Specifically, In terms of deep 
participation learning, there is detailed research from the macro-participation culture to the micro- 
class participation (cognition, emotion, behavior) (Terrenghi et al., 2019; Gee et al., 2019). In terms 
of advanced learning strategies, scholars are committed to exploring various strategies to achieve 
deep learning, from peer teaching as a teacher to self-perception reflection (Nelson et al., 2014).. 
In terms of the development of high-level knowledge and skills, many literatures involve research 
on key compentencies such as creativity (Turvey, 2006) and critical thinking (Wang et al., 2015), 
while Figure 2 shows that it has a distant relationship with deep learning. In terms of transfer 
applications, there are few related studies, and the few existing mainly focus on the transfer of 
knowledge (Green et al., 2013; Nielsen, 2016): Figure 2 shows that transfer is on the edge of 
extreme remoteness confirms this conclusion. 


3.The Concept Evolution of Deep Learning 


At present, deep learning is more than a learning method used to understand basic knowledge, 
it usuallyrefers to learning strategies for obtaining advanced knowledge and its transfer. 
3.1.The change of ideas 
3.1.1.From pursuing understanding to pursuing transfer 

In the early days, the deep learning proposed by Marton and Salj6 (1976) aimed at learning 
approaches or strategies. Simply put, learning by understanding is deep learning, but learning by 
reproduction is shallow learning. Deep learning at this time is reflected at four aspects, they are 
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seeking meaning, connecting ideas, using evidence, and being interested in opinions. Shallow 
learning is mainly embodied in three aspects: information is memorized irrelevantly, learning is 
limited to the syllabus, and adopting minimal effort to avoid failure (Tait, & Entwistle, 1996). This 
kind of learning only generates a limited understanding, little connection between concepts. On 
this account, the famous scholar Ramsden (2003) believed that shallow learning is at best a 
quantity without quality (quantitative change), while deep learning is a cumulative quantity of 
quality (qualitative change). 

It is easy to find that the goal of deep learning at this time is to understand, and the goal of 
shallow learning is to reproduce the test materials. Although shallow learning can produce 
superficial understanding, it is not one of its learning goals. 

Since deep learning starts in a way of understanding and constructing meaning, people soon 
realized that it can lead to better transfer of knowledge and concepts (Van, &Schenk, 1984). After 
entering the 21st century and facing the ever-changing new situation, how to transfer what has 
been learned in the classroom to enable students to succeed in future work and life has become a 
new education challenge. Therefore, the purpose of deep learning has changed from understanding 
to transfer (i.e. learning for transfer), and understanding becomes the basis for deep learning to 
realize migration. The National Research Council of the United States has set the tone of deep 
learning as the process through which an individual becomes capable of taking what was learned 
in one situation and applying it to new situations (i.e., transfer), and the product of deep learning 
is transferable knowledge (National Research Council, 2012). This kind of knowledge includes 
not only the content knowledge of a certain field in the traditional sense, but also the knowledge 
of how, why, and when to apply this knowledge to answer questions and solve problems. 
3.1.2.From focusing on process to focusing on outcomes 

Initially, deep learning was just a kind of learning strategies that Marton and Salj6 advertised 
for students to pursue understanding. Later, this kind of strategies were refined into the deep 
strategies to maximize meaning and deep motivations such as intrinsic interest in the learning 
process(Biggs, 1987). According to Biggs, deep strategies describe the way students are deeply 
involved in tasks. As a prerequisite, this kind of participation has gradually become an integral 
part of the deep learning process (Biggs, Kember, &Leung, 2001). In deep learning, deep 
participation represents the extent to which students are actively pursuing deep learning, and it 
refers to actively participating wholeheartedly. Today, with the emphasis on “student-centered” 
teaching, participation plays an increasingly important role in deep learning. Chinese scholar 
Yunhuo Cui (2017) even directly quoted cognitive participation to define deep learning: it is a 
learning that shows high involvement, high cognitive participation and gains meaning in a complex 
environment. 

Although, after the 1970s, the demand for talents' ability has changed, that deep learning did 
not shift its attention from process to result until the Hewlett Foundation initiated the deep learning 
strategic plan in 2010. Different from the previous deep learning that focuses on the deep 
understanding and basic knowledge transfer, the later deep learning pays more attention to the 
acquisition and transfer of middle-level and high-level abilities. Correspondingly, deep learning is 
also known as a comparative name: deeper learning. 
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Through analysis of the situation at the time, the Hewlett Foundation proposed six deep 
learning abilities that need to be transferred and applied (The William and Flora Hewlett 
Foundation, 2012), they are mastering core academic content, think critically and solve complex 
problems, work collaboratively, communicate effectively, learn how to learn, and develop 
academic mindsets. Soon after, in order to explore the blend of transferable knowledge and skills 
and 21st century competencies, the US National Research Council identified three broad domains 
of competence—cognitive, intrapersonal, and interpersonal, and regarded these three competence 
domains as the three dimensions of deep learning knowledge and skills (National Research Council, 
2012). In fact, the six deep learning abilities proposed by the Hewlett Foundation correspond to 
the three competence domains defined by the US National Research Council, and they both are 
also homogeneous with the capabilities of smart talents, as shown in Table 1. 


Talble 1. The relationship between deep learning abilities, 21st century competencies and smart 
talents capabilities 


competence six deep learning abilities 21st century smart talents capabilities 
domains proposed by the Hewlett competencies 
Foundation 
- key academic -master the basic 
: : subjects -3Rs knowledge proficiently 
mastering core academic : : 
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content _ 
wRiting, and 
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; boa - critical thinking - good at solving complex 
ba think critically and solve e ‘ e E 
cognitive . : and problem problems 
complex problems (including . ; 
; solving - good at judgment and 
the effective use of : : 
; -computing and creation 
professional tools and es 
: digital 
techniques, as well as the : 
oe proficiency 
ability to solve problems ae 
: - creativity and 
creatively) : : 
innovation 
: - good at collaboration 
- collaboration 
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and leadership 
: technology 
interpersonal Tone ae 
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communicate effectively : 
; and media 
(written, oral) : 
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direction 
intrapersonal 


develop academic mindsets 


- career, Civic 
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- ingenuity, good 
personality, pragmatic 


3.2. Complete solution of deep learning concepts 

To sum up, the goal of deep learning has evolved from seeking understanding to pursuing 
transfer. It not only pays attention to the deep participation of students and the strategies students 
adopted in the learning process, but also pays attention to the mastery and transfer of higher-level 
skills. Therefore, this study summarized an elaborate definition of deep learning: Deep learning is 
a meaningful learning method based on understanding and pursuing transfer applications. It 
promotes the development of high-level knowledge and abilities by encouraging students to be 
deeply involved in learning and appropriate use of advanced learning strategies, then realizes the 
application of this knowledge and abilities in new situations or the generation of new high-level 
knowledge and abilities. 

The definition suggests the following features of deep learning, a) deeply involved in learning, 
b) adopting advanced learning strategies, c) focusing on the development of high-level knowledge 
and abilities, and d) based on understanding and pursuing transfer are the four major characteristics 
of deep learning. Among them, deeply involved in learning means fully and actively participating. 
It focuses on student's learning involvement and the state of students’ flow. Advanced learning 
strategy can be judged by "whether it is based on understanding" and "whether it pursues transfer 
application". Considering that deep learning should embody the idea of "student-centered", 
whether the learning strategy is advanced or not also needs to be judged "whether it reflects the 
initiative of the students". 

High-level knowledge and abilities are as shown in Table 1. From the perspective of Bloom's 
taxonomy, it focuses on implementing (1.e., applying what learned in new situations), analyzing, 
evaluating, and creating. The understanding in d) is mostly in-depth understanding. Since 
reflection has been proven to be an important means of in-depth understanding (Haller, Fisher, 
&Gapp, 2007), this understanding can start with deep and repetitive thinking from multiple 
perspectives. Regarding transfer, considering that classroom teaching cannot cultivate all the 
knowledge and abilities to solve unknown problems in the future, this study also takes the 
combination or comprehensive innovation of existing knowledge and abilities as a kind of transfer. 
In this way, the transfer includes both the application of knowledge and abilities in new situations 
and the generation of new high-level knowledge. 


4.The Bottleneck of Deep Learning Research 


Based on our review and analysis, the existing deep learning research basically covered four 
major characteristics of the concept of deep learning. In addition, though the existing studies 
almost regard deep learning as a stable structure-oriented activity process, its flexibility issue has 
not received enough attention. 
4.1.Deep learning demands flexibility 

Whether to encourage students to be deeply involved in learning or to guide them to adopt 
advanced strategies, the ultimate goal of deep learning is to promote the development of students' 
high-order knowledge and ability and its transfer and application. Research has shown that only 
deep understanding can realize transfer applications, and reflection is an important means and 
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effective strategy to realize this understanding, what’s more, some scholars even believe that 
reflection is the only mechanism for deep learning (Svensson, 1977). Reflection requires students 
to be able to review what they have learned, think repeatedly and revise existing ideas at any time. 
In addition, deep learning also needs students to be able to actively select suitable resources, tools, 
and carry out appropriate learning activities according to one's own needs. This personalization 
and initiative can help promote deep learning (Fullan, &Langworthy, 2014). Due to the uncertainty 
of the time for students to review what they have learned, the difference of review contents, and 
the unpredictability of their active choice of resource, tools and learning activities, the structure of 
deep learning activity progress should be diverse and flexible. 

In the flexible progress structure, flexible and interactive self-inquiry guided by teachers is the 
key to the success of deep learning. Self-inquiry can make students believe that they have control 
over the content, methods, and time of learning, and can make them believe that their behavior is 
internally initiated. As a result, it leads to a greater preference for more challenging tasks and a 
greater willingness to put in more effort to understand (Grolnick, &Ryan, 1989), leading to deep 
learning. Research shows that cultivation and transfer applications of the high-level knowledge 
and abilities pursued by deep learning are very difficult for beginners, because they lack the 
schemas that experts use to solve new problems. This can easily lead to students' failure due to the 
lack of knowledge of self-inquiry without flexible interaction between students and teachers 
(Bransford, Brown, &Cocking, 2000). The failure case of deep learning of "learning while writing 
research papers" by Green et al. (2013) proves this point. The flexible interaction between students 
and teachers in self-inquiry are embodied in "individual needs, response whatever is requested, 
and fusion of request and its response", which is different from the teaching mode with clear 
distinction between student-led and teacher-led. 
4.2.The challenge of flexibility 

Flexibility is one of the six challenges of educational development and reform. Deep learning's 
demand for flexibility in the activity progress structure has touched on changes in the top-level 
framework structure, which further increases the difficulty of achieving flexibility. 

In fact, the above flexibility is superficialn, what deep learning really needs is cognitive 
flexibility. This flexible feature is to organize teaching and teach knowledge in different ways, 
prompting students to repeatedly cross-learn (i.e., for the same content, repeated non-linear 
learning is carried out for many times from different perspective at different times and in different 
situations for different purposes). The process of repeatedly cross-learn is accompanied by the 
change of the context and the repetition of de-context and re-context. Context can bridge the world 
of knowledge and the world of life. In detail, de-context establishes a corridor from the world of 
life to the world of knowledge, prompting students to extract knowledge such as laws, trends, or 
common characteristics, and re-context establishes a corridor from the world of knowledge to the 
world of life: every time students solve a problem in a new situation, they will reassemble the 
extracted knowledge and construct the meaning of the current problem. In this way, students can 
form a rich and flexible understanding, and can flexibly apply or assemble relevant knowledge to 
solve problems in changing situations and realize transfer (Jacobson, 1996). This cognitive 
flexibility is a huge challenge in classroom teaching with limited time, heavy tasks and large 
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numbers of people. 
5.Solution Measures 


As to the above flexibility bottleneck, learning architecture may be an effective approach, 
because its outstanding feature is flexibility (MCEETYA, 2003), and it can help understand the 
depth of learning (Scanlan, 2013). 
5.1.Towards a learning architecture 

Shen (2017) provided the idea of transforming pedagogical structure into learning structure, 
considering the existing problems of education and combining the demands of smart learning. 
According to his view, learning structure advocates that students take charge of their own learning 
behavior, which is a unique organizational form about the presentation of learning content, 
organizational sequence, time allocation, self-detection and other elements formed by them with 
the support of specific learning space in order to achieve corresponding learning goals. In the 
learning structure, the main tasks of teachers are to analyze the changes of students' preferences 
and needs, to form diverse resources that meet their individual cognitive habits and their ways of 
representation and presentation, and to construct the support of various learning strategies. It is 
true that the learning structure fully reflects the individuality and initiative of students, and helps 
to promote deep learning (Fullan, &Langworthy, 2014), but it requires too high a student’s learning 
ability and is not suitable for the basic education of primary and secondary schools. More 
importantly, the unique form of learning organization it represents is still concerned with the 
stability of teaching and learning. Therefore, this study believes that teaching needs to move 
towards a learning architecture, so that teaching can not only focus on the individuality and 
initiative of students’ learning like a learning structure, but also cater to the demands of deep 
learning: flexibility. The relationship among pedagogical structure, learning structure, and learning 
architecture are shown in Figure 3. 
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Figure 3 The relationship among teaching structure, learning structure and learning architecture 
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5.2. Smart classroom empowerment 

Fundamentally, the shift to flexibility is driven by the development of technology. In theory, a 
smart classroom that integrates a variety of advanced IT technologies and media devices can 
empower the flexibility of deep learning. So far, although, it has not made deep learning happen 
as we expected. 

As a paradigm of smart learning environment, smart classroom has all the functional 
characteristics of smart environment: seamless connection of learning space, keen perception of 
learning context, natural interaction of learning experience, precise adaptation of learning services, 
full recording of the learning process, and open and integrate of data resources (Zhu, Yu, 
&Riezebos, 2016). These functional characteristics give students more flexibility, effectiveness, 
adaptability, participation, motivation and feedback (Spector, 2014), and empower teachers and 
students to flexibly carry out in-depth teaching and learning. These functional characteristics give 
students more flexibility, effectiveness, efficiency, engagement, adaptivity,and reflectiveness 
(Spector, 2014), and empower teachers and students to flexibly carry out in-depth teaching and 
learning. Specifically, smart classrooms can flexibly create or connect context or real situations, 
and realize the arbitrary change of context required for cognitive flexibility;It can adaptively 
respond to the learning needs of students, and provide the appropriate learning support, 
personalized help and rich media-like learning resource ecology required for repeatedly cross- 
learning. Among them, the recommendation of personalized generative paths will lead to the 
diversity of the activity progress structure. This flexible progress can be fed back to teachers and 
students in real-time in a visual form. 

In addition, smart classroom can pay attention to the learning status of each student, make 
teachers more capable of supervising, guiding and evaluating students, and feel more confident to 
give more initiative to students. This can also stimulate students! interest, motivation and creativity 
to achieve efficient and productive learning and achieve the goal of deep learning outcomes (Li, 
Kong, &Chen, 2015). Yu and Chen (2018) pointed out that smart classroom is a new type of 
classroom form that seamlessly supports deep learning with technology. However, the above 
research status diagram shows that the empowering role of the smart classroom has not been well 
explored: the smart classroom does not appear in Figure 2, and the smart learning environment as 
the upper concept of the smart classroom is only on the edge of extreme remoteness. 


6.Conclusion 


The analysis of this study shows that after more than 40 years of development, the research of 
deep learning has gone through dormant period, germination period, emerging period, and is now 
in the rapid development stage. At present, the research trend of deep learning is to promote 
students to be deeply involved in learning and the appropriate use of advanced learning strategies. 

Two major changes of deep learning occurred in the past 40 years: a)from pursuing 
understanding to pursuing transfer, and b) from focusing on process to focusing on outcomes. 
Morover, deep learning is no longer just a learning approach to understand basic knowledge, it is 
more of a meaningful learning method based on understanding and pursuing transfer applications. 
It promotes the development of high-level knowledge and abilities by encouraging students to be 
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deeply involved in learning and appropriate use of advanced learning strategies, then realizes the 
application of this knowledge and abilities in new situations or the generation of new high-level 
knowledge and abilities. Among them, the generation of new higher-order knowledge and abilities 
is a new aspect of transfer that we advocate. 

However, the analysis revealed that the flexibility of deep learning is currently overlooked, 
while flexibility is the appeal of deep learning. This appeal involves the change of the top-level 
framework structure and requires cognitive flexibility which might be extremely challenging to 
achieve. In this regard, this study suggests that deep learning move towards a flexible architecture 
and try to solve this problem with the help of the enabling of a smart classroom. 
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Creation and Use of Self-made Teacher Videos in the Secondary Classroom 
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Abstract 


The challenges of the 2020 global COVID-19 pandemic have thrust K-12 teachers into 
the world of online and hybrid learning. As a result, many teachers are looking for new and 
innovative ways to provide learning opportunities to students through online videos. The popular 
social media video streaming site YouTube provides a convenient resource for teachers to share 
video content as well as for researchers to observe teaching. This study provides an exploratory 
analysis of two differing approaches to creating and using teacher self-made videos in secondary 
classrooms. The study compares videos that were filmed in class, which document specific 
problem solving, with those made out of class to deliver direct instruction. Descriptive statistics 
provide insight into key comparisons of video length, number of views, and identify areas of 
interest; Distractions, Advertisements, Technology Troubles, Video Issus, and Audio Issues that 
were all found to influence delivery of instruction through online videos. 


Key Words: Teacher Self-made Videos, K-12 Teacher Videos, YouTube, Secondary Education, 
Online Learning 


Introduction 


The COVID-19 pandemic has catapulted learning technology into the forefront of 
education (Bonafini & Lee, 2021; Fackler & Sexton, 2020; Smith & Colton, 2020). Teachers 
have been compelled through circumstances to adopt technology-based learning approaches that 
many were reluctant to accept and implement prior to the pandemic (Francom et al., 2021). 
Teachers are quickly moving to create rich learning opportunities to address the needs of their 
students and provide support to parents as at-home learning facilitators (Archambault & Borup, 
2020). This includes the use of videos in both synchronous and asynchronous settings. 
Lowenthal and Covey (2021) observe that video is an effective instructional tool that is also well 
suited for conducting education research. The purpose of this study is to provide an objective 
review of these teacher-made videos to better understand the varied approaches to both creating 
videos and how they are used to deliver or support instruction in the K-12 secondary classroom. 


Literature Review 


Anecdotal observations of instructional videos used in the classroom prior to the 
pandemic were primarily limited to those produced for educational use by outside organizations. 
Videos used in the classroom were selected based on their relevance to the topic being taught and 
sometimes lessons were formed to fit around the video. As more teachers move to online and 
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hybrid models of learning, it has become more difficult to utilize existing videos to meet the 
needs of all students. Innovative teachers have turned to creating their own instructional videos 
to deliver content to their students, which may be attributable to the extensive use of video in 
teacher preparation programs (Hollingsworth & Clarke, 2017). Lowenthal and Carvey (2021) 
observed that “despite the increased use of video in teacher education, questions remain about 
effective ways to use video in online video-based instructional modules” (p.225). 


The Challenge of Rapid Transition 


The sudden shutdown of schools and the government imposed at-home quarantines 
which necessitated the abrupt jump to distance learning revealed how unprepared many 
educators are to provide technology-based learning (An et al., 2021; Francom et al., 2021; 
Hodges et al., 2020). Despite the prevalence of technology in society and the classroom, many 
teachers and administrators continue to rely upon traditional models of instruction that are 
dependent upon location and presence (Smith & Colton, 2020). While online learning can offer 
many affordances, the transition from theory to practice can present unique challenges in normal 
times and even more so during a global pandemic (Hodges et al., 2020; Johnson et al., 2019; 
Tawfik, 2021). An et al. (2021) observed that “many schools and teachers were not well prepared 
for the sudden move to online teaching due to the pandemic” (para. 2). Using videos can help 
teachers draw students into the learning environment regardless of their physical location to 
facilitate learning (Di Paolo et al., 2020). 


The use of videos in the K-12 classroom has been a common practice for decades 
(Aranya, 2020). Today more teachers are leaving textbooks behind in favor of digital resources 
that include high quality videos (Blomgren, 2018). Indeed, many teacher education programs 
now actively incorporate the use of videos in a variety of ways to prepare teachers for the 
classroom (De Voto & Thomas, 2020). The shutdown of schools due to the COVID-19 pandemic 
can be seen as “an unprecedented crisis and differs from other major school emergencies, such as 
school shootings and bomb threats” (An et al., 2021, para. 2). Hodges et al. (2020) noted that 
“well-planned online learning experiences are meaningfully different from courses offered online 
in response to a crisis or disaster” (para. 1). 


YouTube: Working in a Familiar Space 


YouTube is a global social media website where users share original videos. According 
to the YouTube website, “millions of people come to YouTube to be informed, inspired, or just 
plain delighted” (YouTube, n.d.). While YouTube is a popular media platform, Di Paolo et al. 
(2020) state that “technology alone does not motivate students to learn; however, instructors can 
use technology purposefully, effectively and efficiently to enhance learning (p.452). Lowenthal 
and Covey (2021) also identify video as a powerful media format that “can show things in ways 
that previously were not possible” (p.233). Teaching under what are considered normal 
conditions can be a complex process (Cuenca & Zaker, 2019). Because of the social restrictions 
created by the COVID-19 pandemic, both parents and education leaders have increased their 
demands on teachers to provide and promote effective technology-based learning opportunities 
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(Bonafini & Lee, 2021). Fackler and Sexton (2020) observe that the current climate creates a 
“complex nature of teaching in these uncertain times” (p. 11). 


Lingering Questions Remain 


Questions remain around what constitutes a good video and why teachers choose to use 
existing videos or make their own videos for their classrooms (Wijnker et al., 2019). More 
information is needed to help educators understand “details on how video is recorded and 
selected when developing video-based instructional interventions” (Lowenthal & Cavey, 2021, 
p.226). Di Paolo et al. (2020) identified four significant areas of consideration as “planning, 
development, delivery and reflection” in the meaningful use of online videos for distance 
education (p.459). Circumstances for learning and educational technology are continually 
changing and evolving, therefore repeated evaluation of ongoing efforts are needed to ensure that 
students continue to receive the education they both need and deserve. 


Methodology 
Purpose and Participants 


The purpose of this study is to add to the understanding of how teachers are using self- 
made videos to teach students as a result of the COVID-19 pandemic’s influence on education. 
Specifically, the study compares publicly available videos on YouTube from two secondary 
teachers with differing approaches to classroom video creation and use. This study utilizes a 
nonrandom, purposive sample of teacher self-made videos from two teachers who demonstrate 
differing approaches to creating and using video-based instruction in the classroom. Both 
teachers began posting videos regularly on YouTube in August 2020. This study reviews videos 
posted over the fall semester from August 2020 through December 2020. 


Teacher | is a secondary math teacher from a rural school district with approximately 
2,000 students. Teacher 1’s primary approach to creating videos is through in-class recordings 
that are uploaded for students to review after class or at home asynchronously. Teacher 1’s first 
YouTube video was a virtual open house that posted August 19, 2019 and was not part of the 
study. The next video posted by Teacher 1 was on August 19, 2020, when Teacher | began 
posting videos regularly that corresponded with work done in the classroom. 


Teacher 2 is a secondary biology teacher in a district of approximately 5,000 students. 
Teacher 2’s primary approach to creating videos is through prerecorded lectures that are 
uploaded for students to view before, during, and after class, or asynchronously for at-home 
students. Teacher 2’s first YouTube video posted on September 14, 2020, followed by 
subsequent videos that were reviewed as part to the study. 


Materials, Data Collection, and Procedures 


This study utilized commercially available internet enabled computers with internet 
browsers to view online YouTube videos. Data was collected from both teachers on videos that 
were posted between August 2020 and December 2020. Videos were selected from each teachers 
YouTube channel by selecting the videos tab and then sorting the videos from oldest to newest. 
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Videos were watched in the order they were posted. Each video that was posted during the time 
frame of the study was selected and viewed from start to finish. The video play back speed for 
most videos was set between 1.5 to 2.0 to facilitate efficient review. Where clarification was 
needed, the video speed was adjusted to normal. Collected data was stored in a spread sheet and 
backed up on a portable storage drive. Built in formulas within the spreadsheet software were 
used to calculate descriptive statistics. 


Results 


Over the course of the study, both teachers posted videos regularly on YouTube. Teacher 
1 posted more frequently, with 152 total videos, than Teacher 2, with 32 total videos. The total 
length of combined videos for Teacher 1 was 20 hours and 9 minutes. While Teacher 2 recorded 
and posted a total of 7 hours and 47 minutes. The average video length for Teacher 1 was 7.95 
minutes and for teacher two it was 14.49 minutes (see Table 1). 


Table 1 
Total Videos | Total Time of Videos Average Video Length 
Teacher 1 152 20 hours 9 minutes 7.95 minutes 
Teacher 2 32 7 hours 47 minutes 14.49 minutes 


The key difference to the creation of videos between Teacher 1 and Teacher 2 was 
whether the videos were made in class or out of class. In class videos are videos made with 
students present, and out of class videos are videos made without students present. Teacher 1 
made 82.24% of videos in class and 17.76% of videos out of class. Teacher 2 made 6.25% of 
videos in class and 93.75% of videos out of class (see Table 2). There were also significant 
differences between the number of views each teacher received for their videos. Table 2 shows 
that Teacher 1 received 1,336 total views for an average of 8.78 per video while Teacher 2 
received 5,719 total views for an average of 178.71 views per video. 


Table 2 
In Class Out of Class Total View Average Views 
Teacher 1 82.24% 17.76% 1336 8.78 
Teacher 2 6.25% 93.75% 5719 178.71 


During the viewing of videos several important areas of interest emerged, specifically 
Distractions, Advertisements, Technology Troubles, Video Issus, and Audio Issues. Distractions 
were identified as anything that took away from the lesson or teaching being presented. 
Advertisements were separate from the video content and part of the YouTube streaming 
platform. Technology Troubles were any issues associated with technology devices not working 
correctly on camera. Video Issues were typically glitches in recording that could not be 
attributed to user error. Similarly, Audio issues were issues with audio feed that could not be 
attributed to user error. Table 3 shows the percentage of videos for each teacher according to 


these categories. 
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Table 3 


Distractions | Advertisements | Technology | Video Issues | Audio Issues 
Troubles 
Teacher 1 26.32% 0.00% 37.50% 31.58% 36.84% 
Teacher 2 21.88% 46.88% 3.13% 31.25% 31.25% 
Discussion 


An initial overview of the data collected for both teacher’s reveals a large difference 
between the total number of videos posted by each. Teacher | posted nearly five times more 
videos than Teacher 2. This is explained through data collected from the videos and provides 
insight into why each teacher may have chosen their particular approach to creating videos. 
Teacher 1 created and posted videos for at least two classes and possibly a third. Because the 
videos were posted to YouTube and then linked to assignments in Google Classroom, it is not 
readily clear which videos went with which classes however, it is clear that Teacher | created 
videos for a Fundamentals of Math and a Geometry class. Teacher 2 created videos for a high 
school biology class, and as observed in the videos often recorded during the teacher conference 
period. Having one lesson to prepare is quite different than having two lessons to prepare which 
may explain why Teacher 2 chose to create videos prior to class and Teacher | out of necessity 
and a lack of time created videos in class. This may also help to explain why Teacher 1’s videos 
were on average only about half as long as Teacher 2’s videos. Whereas Teacher 1 focused on 
delivering direct examples of how to work problems, Teacher 2 used the video for direct 
instruction. 


Considering the circumstances of each teacher also helps to explain the percentages of 
videos that were made in class vs out of class. Noticeably, Teacher 1’s out of class videos were 
made while the teacher self-reported to be under personal quarantine or while the school was 
physically closed, and students participated in at-home remote learning. No clear explanation 
emerges for why Teacher 2 chose to create some videos in class with students present, though it 
should be noted that some of the out of class videos were made from home while the school was 
also closed, and students were switched to remote learning. Notably both schools, from different 
school districts, experienced physical closures and forced distance learning due to the COVID-19 
pandemic during the study period. 


When looking at the average number of views per video there is a dramatic difference 
between Teacher | and Teacher 2. Teacher 2 had nearly twenty times more average views than 
Teacher 1. This perhaps due to the way the videos were used in class, however without access to 
specific viewer data it cannot be clearly stated why Teacher 2 had so many more average views 
than Teacher 1. However, it is reasonable to conclude that their approaches to using the videos 
contributed to more views for Teacher 2. Teacher 2 used the videos as lectures for students, and 
also included comments to non-students, which may be partly why the general nature of the 
informational videos may have attracted a larger audience. Teacher 1, who used the videos to 
document working specific problems in class may have received fewer views because of the 
specific nature of the content as well as the redundancy for students that had already seen the 
problems worked in class. 
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The information gathered in Table 3 provides perhaps the greatest insight into a 
comparison of the two approaches to classroom video creation. Distractions ranged from 
classroom announcements during both teacher videos, to off task students, and the teacher 
engaging in sidebar conversations. Noticeably, the percentage of videos with distractions was 
closely similar with both teachers. Teacher 1 had no advertisements before, after, or during the 
posted videos so it came as somewhat of a surprise when almost half of Teacher 2’s videos had 
at least one and often multiple advertisements associated with them. Further review revealed that 
Teacher 1’s account was marked by YouTube as child related so comments were disabled and a 
link for YouTube kids was at the bottom of the page. Teacher 2’s account was not marked for 
children, so the content was not limited. The presence of advertisements on required videos 
creates ethical questions and an opportunity for further research and discussion. Technology 
troubles were observed as both teachers struggled with their own issues. The increased number 
of Technology Troubles experienced by Teacher | is partially attributed to the fact that the 
videos were created during live class sessions and presented little opportunity for error. A simple 
glitch for Teacher | affected both the recordings and the real-time student instruction. Recording 
outside of class time may explain why Teacher 2 had significantly fewer Technology Troubles. 
Quality of the recording equipment became a factor for both teachers, as both experienced a 
similar percentage of issues. Video issues for Teacher 1 were related to incorrect orientation, 
failure to stop and start when intended, and video skips that created gaps in instruction. Video 
Issues for Teacher 2 were related to the quality of the camera and the lighting. Teacher 2 often 
lectured in front of a smartboard that created back lighting and shadows as well as issues of 
keeping the content on the smartboard in frame and clearly visible to students. Audio Issues for 
Teacher | were related to the difficulty of recording all of the audio in a classroom with a single 
microphone. Videos posted by Teacher 1 were filled with one sided conversations because most 
of the student’s comments and questions could not be understood. This finding seems to support 
Ferdig and Kosko (2020) observation that the physical proximity to the camera can provide an 
advantage to those who are closer and a disadvantage to those who are further away. Audio 
Issues for Teacher 2 were attributed to movement by the teacher while lecturing as the audio 
would fade in and out as the teacher faced the board and then turned to face the camera. 


Considerations for the Future and Conclusions 


Understanding how these videos are created and used in a way that is both efficient for 
the teacher and effective for the student is critical. Additional research has concluded that teacher 
made videos may offer help in “understanding the complexity of classroom teaching but also 
builds spaces for reflective thinking and learning through practice” (Wetzel et al., 2017, p.535). 
As required changes begin to ease, it will fall more to education leaders to set a tone for the 
continued use of tools deemed essential during the pandemic (De Voto & Thomas, 2020). 
Farmer and West (2019) conclude that “as new online programs emerge and existing programs 
continue to develop, policies and practices at an organizational level should be established only 
after careful consideration of their impact on teachers” (p.116). The COVID-19 pandemic has 
forced the rapid creation of technology-based learning content. Helping teachers understand the 
pros and cons of various approaches to media creation will help them make more informed 
decisions about how to best serve their students. “Teacher educators must find real-world 
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opportunities to develop technology-based instructional experiences for teachers to develop 
technology competencies” (Smith & Colton, 2020, p.454). 


This study has sought to provide both current and future teachers with a better 
understanding of issues associated with two approaches to creating videos for classroom use. 
Teachers who create videos for classroom use must decide when and how they will create the 
video and how the video will be used to support learning. The shutdown and subsequent 
measured reopening of schools has forced teachers to consider new ways of educating students. 
As more teachers turn to creating their own video content consideration is naturally given to how 
these videos will be made and what purpose the videos will serve in the classroom. Lowenthal 
and Cavey (2021) observed that “as it becomes easier and easier to create, edit, and share video, 
educators, instructional designers and curriculum developers, and researchers will continue to 
experiment with ways to intentionally use video to improve teaching and learning” (p.233). 
Recording videos in a live classroom setting provides relevant examples for students to draw 
upon as they recall the experience. However live classroom recordings are also subject to 
disturbances and distractions that teachers may not have time to edit out. Recordings that are 
made ahead of time offer teachers a more thought-out opportunity to deliver direct teaching, yet 
it can be time consuming to prepare, create, and then post a video. Videos made prior to class 
however do offer greater flexibility when considering how they will be used. Teacher made 
videos offer more specific instruction to students from a relevant and reliable resource. One 
question that remains is whether technology practices that were implemented due to the extreme 
circumstances of the global pandemic will continue as part of the new normal or fade away into 
old habits (Tawfik, 2021). 
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Abstract 


In recent years, the educational strategy called the flipped classroom (FC) has gained 
notable publicity and popularity in education. A substantial number of articles related to FC have 
been published aimed to synthesizing this extensive body of research on the flipped classroom. 
Hereto, this paper examined 13 reviews aiming to identify the characteristics of review studies 
and the current state of knowledge on the flipped classroom as a pedagogy approach. Despite the 
methodological differences, the results of this review confirm that FC has positive impacts on 
students’ learning experiences in general. This study revealed a need for increased support for 
research in FC design, implementations, and practices. More longitudinal analyses are necessary 
to better understand the best practices of FC implementation or whether the effectiveness of FC 
practices is influenced by contextual elements, such as disciplines, subject domains, the 
complexity of learning content, academic levels, and implementation duration. 


Keywords: flipped classroom, inverted classroom, scope review, comparison study 
Introduction 


Flipped classroom (FC) is an instructional approach that extends traditional classroom- 
based learning through technology (Karabulut-Ilgu et al., 2018). FC is also referred to as inverted 
learning or flipped learning. The concept of FC emerged in 2000 when Lage, Platt, and Treglia 
proposed the idea of an “inverted classroom”. In an inverted classroom, the students’ first 
learning activity happens before the class and then uses the in-class time for more engaging 
activities like problem-solving (Lage et al., 2000). In 2007, FC gained its popularity after two 
high school teachers, Jonathan Bergmann and Aaron Sams, used video lectures as students’ pre- 
class assignments for their chemistry classes (Lage et al., 2000; Tucker, 2012). Various models 
were proposed for FC implementation, such as the FLIP model (i.e., flexible environment, 
learning culture, intentional content, and professional educator) (Flipped Learning Network, 
2014) and “FLIPPED” model, which extends the FLIP model by adding three more elements 
(i.e., progressive activities, engaging experiences, and diversified platforms (Chen et al., 2014). 
While there is no fixed model, the essential idea of FC is to have lectures and direct instruction 
delivered before class through online videos. And then, the actual class time is used for more 
engaging collaborative learning activities, such as hands-on activities, discussions, and 
interactions (Flipped Learning Network, 2014; Tucker, 2012). The instructional approach aims to 
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transform students from passive learners to active learners, who can self-direct their learning to 
improve personal analytical, integration, and critical thinking skills for higher levels of learning 
(Hu et al., 2017). 

Given the advancement of technology that provides tools for delivering flipped learning, 
FC has gained much popularity and scholarly attention in recent years (Akcayir & Akcayir, 
2018; Bergmann & Sams, 2012). Various class activities, teaching tools, and techniques have 
been used to carry out the FC approach (Lundin et al., 2018; O’Flaherty & Phillips, 2015; Ward 
et al., 2018). Examples of the inside- Fand outside-of-class activities include reviewing case 
studies, playing games, role-playing, working on peer-reviewed assignments, watching video- 
recorded lectures, and working on post-class homework (Ward et al., 2018). The teaching 
techniques used in FC include prerecorded lectures, case-based presentations, team-based 
discussions, panel discussions, and debates (O’Flaherty & Phillips, 2015). Digital technology is 
also used to implement FC. For example, social media or networking sites like YouTube or blog 
posting is applied as platforms to share educational resources (Lundin et al., 2018). Additionally, 
the FC approach is widely implemented across different education levels, such as K-12 and 
higher education (Cheng et al., 2019), and disciplines, like engineering (Kerr, 2015), clinical 
pharmacy (See & Conry, 2014), chemistry (Baepler et al., 2014), and computer science 
(Giannakos et al., 2014). 

The advantages of FC included better students’ learning achievements, improvement on 
learning experiences, and increases students’ motivation and satisfaction during the learning 
process are broadly indicated in previous research (Koo et al., 2016; Sergis et al., 2018). On the 
other hand, the challenges of FC included time limitation for pre-class activity preparation, time- 
consuming and less effective for learning (Koo et al., 2016; Roehling et al., 2017). 

For a field to continuously progress, it is essential to be aware of its developmental 
patterns in the past to obtain insights for future implications (Dwivedi et al., 2011). As FC has 
progressively expanded over the years, a lot of research addressed the development of FC and its 
impacts. However, each review has its own focus, scopes, and comprehensiveness (Karabulut- 
Ilgu et al., 2018; Oliver & Luther, 2020). For instance, Cheng et al. (2019) reviewed the impacts 
of FC on students’ learning outcomes. Liu and colleagues’ (2019) review focused on the 
teaching effects of FC. Some research addressed FC and its impacts on both teaching and 
learning (Karabulut-Ilgu et al., 2018; Oliver & Luther, 2020). 

Furthermore, mixed results were reported in the review studies with a focus on FC 
effectiveness. Researchers indicated that the positive impacts of FC on student learning, which 
included better learning performance, enhanced student engagement, increased student 
satisfaction, the development of problem-solving and cooperation skills, and improvement of 
theoretical knowledge (Akcayir & Akcayir, 2018; Hu et al., 2018; Karabulut-Lieu et al., 2018). 
On the other hand, although research results showed that FC in class seemed to produce positive 
learning outcomes, the effect is not long-lasting and not all course evaluations are positive 
(Betihavas et al., 2016; Presti, 2016). 

With the different perspectives of examination of each review, different sets of studies 
might be selected, and, accordingly, different analysis results and conclusions would generate. 
Despite the contribution of these researchers, the diversity in selected information in existing 
review articles prompts a need to conduct a review of reviews study to provide a synopsis of the 
evidence of FC. Reviews of reviews are referred to as umbrella reviews, reviews of systematic 
reviews, overviews of reviews, a synthesis of reviews, and a summary of systematic reviews 
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(The Joanna Briggs Institute, 2014). This approach has been used to summarize the extensive 
scientific knowledge on widely explored research topics in nursing and medical fields (Egan et 
al., 2008; Mikton, & Butchart, 2009) and is becoming increasingly common in education and 
other fields (Chen et al., 2018; Pahlevan-Sharif et al., 2019). Reviews of reviews aim to 
summarize the existing research syntheses regarding a topic of interest (The Joanna Briggs 
Institute, 2014), which is compatible with the purpose of this current study. 

The purpose of this study is to provide an overview of the features and impacts of FC 
from previously published FC reviews in the past decade. Through the lens of reviews of 
reviews, the close investigation of relevant FC reviews can provide vital insights into the 
expanding and popular trend of FC and to identify the insights to shed light for future research. 
This study is intended to capture the essence of the existing reviews in the area of FC to answer 
the following research questions: 


1. What are the characteristics of review studies on the flipped classroom? 
2: What are the findings of the FC reviews? 
Methodology 


Standard reviewing methods were applied in the current study to identify existing 
reviews, instead of primary research (Khan et al., 2001). The present study uses reviews as the 
unit of analysis, which differs from a conventional review that uses primary studies as the unit of 
analysis (Kache & Seuring 2014; Keller & Torre, 2015). 


Search Strategy 


Social Science Citation Index (SSCI) was the database used to identify the relevant 
reviews on flipped classrooms. SSCI was selected because it covers top-tier or highly ranked 
journals in social sciences based on the sophisticated selection process, specifically, the Web of 
Science Core Collection Journal Selection Process (Web of Science Group, 2019). The Web of 
Science (WoS) platform, one of the largest academic database search platforms, was applied to 
access SSCI. 

In order to identify the largest possible number of articles in our initial search, the authors 
of this study used strategies that included a range of terms for their main research variables (e.g. 
flipped classroom, flipped approach, flipped instruction, flipped learning strategies, inverted 
classroom, systematic review, meta-analysis, scoping review, etc.). The Boolean operator “or” 
was used with related terms to broaden their search before narrowing down by the selection 
criteria. They also used truncation to include variation of words. For example, the authors used 
“flip*” to find items, including terms like “flip,” “flipping,” and “flipped” and “‘invert*” to 
search article items that contained “inverting” or “inverted.” After completing the above search 
process, they further applied selection criteria to screen out irrelevant results. The language 
restriction was English. 
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Inclusion and exclusion criteria 


Criteria for inclusion of papers included (a) peer-reviewed journal articles; (b) articles 
classified as reviews, systematic reviews, or meta-analyses; (c) articles that reported on/about the 
flipped classroom; and (d) articles that were published in English. Studies were included if they 
reviewed any studies on the flipped classroom. Studies that did not meet these inclusion criteria 
were excluded, such as those that reviewed conference papers instead of published journal 
articles, studies that did not follow the systematic analysis procedure, and those that failed to 
report the process of literature selection. Restricting eligibility to publications in peer-reviewed 
journals enabled consistent quality across disciplines and databases covered in the literature 
search. 


Selection process and data analysis 


After completing the initial search process, all studies were examined by the two authors. 
After the initial database search, the two authors independently scrutinized the titles and abstracts 
from the electronic searches followed by the full text of all citations that definitely or possibly 
met the predefined selection criteria. If a disagreement occurred between the two authors, a third 
person would share their judgment. A discussion would follow among the three individuals until 
a consensus was reached. This exact process was also applied when a disagreement occurred 
during the coding stage. 

After excluding duplicated records identified across SSCI searches, the titles and 
abstracts of 361 records were screened. After a full text review and discussion, a total of 13 
unique reviews met the inclusion criteria. The two authors assessed the manuscripts 
independently and resolved any disagreements about inclusion by consensus after discussion. For 
each study included, information was obtained, such as bibliographical data, study purpose, 
study design, learning context, and primary results. Content analysis is the approach for data 
analysis. Both authors coded the qualitative data attained from the articles, respectively. Any 
discrepancies in the coded data between the two authors were discussed with a third reviewer 
until reaching a consensus of the coded data. 


Results 
Characteristics of the Included FC reviews 


This section discusses the characteristics of retained FC reviewed, including background 
information (i.e., publication year, academic discipline, education level), research focus, and 
research methods. 

Background information. Results show that one to two FC review research were 
published in 2015, 2016, 2017, and 2019. However, in 2018, the FC reviews publications peaked 
when seven articles were published that year. Additionally, authors of the 13 FC reviews came 
from a variety of regions, including North America (five articles from the U.S.), Asia (two from 
Hong Kong, one from China, and one from Taiwan), Europe (one article from Turkey and one 
article from Sweden) and Australia (two articles). All of the FC reviews focused on English- 
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based FC studies. Nine of the 13 FC reviews examined FC studies within a single academic 
discipline, including nursing (4), health (2), engineering (1), mathematics (1), and education 
technology (1). Moreover, the majority of them examined the effectiveness of FC in the context 
of higher education. 

Research focus. When examining the purpose study of the included FC reviews, four 
different research focuses emerged, including overall trends, effectiveness of FC on learning, 
current state of FC knowledge or practice, and benefits and challenges of FC implementation. It 
is worth noting that some of the included FC reviews may have had more than one study focus 
and/or was conducted in multiple stages. For example, the review of Lo et al. (2017) initially 
conducted a meta-analysis of 21 comparison studies about FC to examine the effects of FC for 
math education learning, following synthesized findings of 61 FC studies to identify the benefits 
and challenges of FC on students’ learning. 

Some features were found among the FC reviews with the same research focus. For 
example, the FC reviews that focused on FC current development situation frequently applied 
some aspects to present the developmental trend among the FC studies, such as publication trend 
(i.e., year, author, publication venue), keyword, and subject area. The FC reviews that focused on 
FC effectiveness examined the issues and factors related to the effect of FC approaches on 
teaching and learning, such as students’ academic levels and duration of implementation. 
Furthermore, some of the FC reviews that addressed the current knowledge or practice status of 
FC also investigated the development or effectiveness of FC. For example, Ward’s (2018) 
review integrated the application of FC in nursing education based on 14 related literatures. In 
addition to identifying main FC application themes in nursing education, students’ academic 
learning outcomes and their measurements were also provided. 

Research methods. Research methods of FC reviews can generally be categorized into 
qualitative and quantitative. The qualitative methods focused on content analysis to generate 
common themes among studies, and the quantitative methods studies focused on numerical data. 
However, some qualitative studies may have used quantitative data-analysis techniques, such as 
frequency or percentage analysis to describe the themes. Therefore, the authors broadly sorted 
FC reviews into three categories of research methods: qualitative review, quantitative review, 
and mix-method review. Among the thirteen FC reviews, one of them used a mixed method 
(8%), seven used a qualitative method (54%), and five of them used quantitative methods (38%) 
to analyze data. In terms of research approaches, there were meta-analyses, systematic review, 
literature reviews, and scoping review. All qualitative reviews utilized content analysis. 
Furthermore, FC reviews indicated that techniques and standard content analysis were used for 
qualitative data analysis, such as the coding protocol, rating scale, and framework. Some 
included studies that applied more than one research approach. For example, the study of Lo et 
al. (2017) stated that they applied two approaches, such as content analysis and meta-analysis. 
The content analysis was used to examine current FC practice, and the meta-analysis method was 
used to conduct the statistical data of the effect value by homogeneity studies. 


Primary Findings of FC Reviews 


Three overarching themes emerged from results of FC reviews, including overall trends, 
effectiveness of FC on learning, and benefits and challenges of FC implementation. 
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Research trends of FC studies. The top two main trends of FC studies in this FC 
reviews were (a) focused more on higher education and (b) small-scale research. Higher 
education was the development trend that was indicated most frequently in eight of the thirteen 
included FC reviews. For example, Akcayir and Akcayir (2018) pointed out a particular 
composition of FC studies participants. While the FC approach seemed suitable for courses in all 
learning levels, many study participants in existing FC studies are composed of students in 
higher education institutions. Moreover, Cheng et al. (2019) pointed out that accessibility of 
study participants is also one of the reasons for this, since FC researchers are mainly from higher 
education institutions. It might be easier for them to find access to student populations in higher 
education than in K-12 schools. 

Small-scale research was also frequently used by FC studies, as it was indicated in half of 
the included FC reviews. Although the length of time of FC implementation varied considerably 
study-by-study, seven of the thirteen reviews revealed that many of the FC studies were small- 
scale research, such as small sample size, localized recruitment, and descriptive design. The 
duration of the implementation mostly ranged from one unit in a semester to one full semester. 
The tendency of localization in FC studies might result in the application of small-scale design. 
For example, 26 of the 31 FC articles that Lundin and colleagues (2018) reviewed were 
composed of experiences from a single course or classroom. 

Effectiveness of FC on learning. Results showed that both qualitative and quantitative 
analysis methods were used to examine effectiveness of FC on learning. Table 1 summarizes the 
study findings regarding the effectiveness of FC that is indicated in the included FC reviews. To 
summarize, 12 of the 13 FC reviews concluded that FC had overall positive impacts on learning 
compared to in-class, lecture-based learning. It is also worth noting that Chen et al. (2018) found 
a progressive improvement tendency in the FC outcomes over time through a meta-regression 
analysis. In other words, FC studies published in recent years seemed to support the improved 
outcomes under FC conditions more. Moreover, the effectiveness of FC was usually evaluated 
through academic performance (n=4), satisfaction (n=5), engagement (n=4), motivation (n=1), 
and self-reported learning attitude (n=1). 


Table 1 
Summary of Findings Regarding the Effectiveness of FC 
Category n Source of Review 


Overall positive impact on student learning 12 S1, 82, 83, S4, S6, S7, S8, 


outcomes S9, S10, S11, $12, $13 
Academic performance (e.g. test scores, 4 82, $3, $4, S8 
course grade, knowledge score) 
Satisfaction $3, $9, S10, S11, S12 
Engagement 4 $3, S6, S9, S11 
Motivation 1 S3 
Attitude 1 S3 

Factors on the effectiveness of FC 
Type of content knowledge 5 S1, 88, S11, $12, S13 
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Subject areas/discipline 3 S1, $4, S13 
Student levels (K-12, HE) 2 S1, 83 


Duration of Implementation 2 S1, 83, 


Some FC reviews further examined influential factors associated with the differences in 
the positive effects of FC on learning and found possible factors, including content knowledge 
(n=5), subject area (n=3), student academic level (n=2), and duration of implementation (n=2). In 
terms of knowledge content, many FC reviews concluded that FC had greater impacts on 
improving practical knowledge learning, such as design and experiment than theoretical 
knowledge learning (Hu et al., 2018). 

Results also indicated that the variance of FC effectiveness in learning was within 
different subject areas. For example, Cheng et al. (2019) found a higher effect size of FC on 
learning subjects like art and humanities than social sciences, mathematics, health, etc. Chen et 
al. (2018) indicated that the FC approach had greater positive application impacts on medicine 
and pharmacy than nursing and other health-related professions. The difference may be 
reasonable, since the FC approach did not seem applicable to certain types of content knowledge. 
Therefore, it was necessary to consider the subject area’s features when the FC approach was 
applied. Additionally, the effects of FC on learning outcomes among different student's academic 
levels were not conclusive. Akcayir and Akcayir (2018) indicated that the effects of FC on 
learning outcomes were higher in older learners. However, Cheng et al. (2019) found that the 
impact of FC approach outperformed the non-FC approach in higher education, but the effects on 
graduate students were overall negative. This means that postgraduate students learned better in 
non-FC conditions. The duration of FC implementation may result in differences in the FC 
effects, and a shorter implementation period seemed better. Cheng et al. (2019) found that 
overall, while students in FC were significantly outperformed students in non-FC conditions, the 
FC studies with a longer duration of FC implementation had smaller effect sizes than studies 
with a shorter period. Akcayir and Akcayir’s (2018) study also indicated a similar conclusion 
that the effects of FC might not sustain over longer-term applications. 


Benefits and challenges of FC implementation. The benefits of FC implementation 
commonly reported by both students and instructors were flexibility, it enabled individualized 
learning, and students were better prepared before class (Akcayir & Akcayir, 2018; Karabulut- 
Lieu et al., 2018). In addition to the benefits of FC implementation, challenges of FC 
implementation were noted in the FC reviews. For instance, there were concerns about the 
quality of the learning materials, the time-consuming nature of FC for course content 
preparation, technological issues, the need for out-of-class activity guidance, and the time 
required by instructors to integrate into their curriculum. Some FC reviews also indicated 
possible conditions that hindered instructors from adopting the FC approach. This was especially 
true regarding courses that involved large amounts of factual content and hand-on activities as 
well as the requirement of frequent interactions and collaborative group activities (Akcayir & 
Akcayir, 2018; Karabulut-Lieu et al., 2018; Lo et al., 2017). 
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Discussion & Conclusion 


This review of reviews included 13 studies about FC that were published by SSCI- 
indexed journals in the last decade. The results of FC reviews raised some concerns about the 
existing FC studies from the following aspects, including the study design of FC studies, the 
definition of learning effects, and the analysis of FC effectiveness. 


The Study Design of FC Needs to Be More Diverse 


Regarding the study design in investigating FC practices and evaluating FC effectiveness, 
there were several concerns identified by FC reviews. First, the majority of FC studies focused 
on comparing FC and general non-FC conditions. However, as Lundin et al. (2018) indicated in 
their review, comparison studies' fundamental problem was in these kinds of studies. This means 
that the non-FC context, like the in-class, lecture-based teaching approach, was most likely to be 
treated as unsuccessful, and FC was seen as a solution. The effect of FC in improving student 
learning seemed to be taken for granted, commonly without explanation of the pedagogical 
design and implementation settings (Lundin et al., 2018). Therefore, future FC studies need to 
employ more alternative quantitative, qualitative, and mixed methods to understand the FC 
phenomena in depth. 

Second, many FC studies evaluated influential factors based on small-size, local case, 
and short implementation period. For example, 26 of the 31 FC studies that Lundin et al. (2018) 
included in their review were locally situated in terms of the sample (commonly 20 to 40 
students) or case (mostly in higher education) and focused on only one course. Such a small- 
scale design made the generalizability of these studies' results a likely issue. The fragmented 
knowledge contributions also showed that the development of FC's related field had yet to be 
stabilized (Lundin et al., 2018). Moreover, FC's positive effect at the beginning might not be 
sustained throughout the semester when students in non-FC conditions catch up by the end of the 
course (Evans et al., 2018). In order to see if the FC approach can be appropriately adapted in a 
broader-level implementation, higher quality studies focusing on longitudinal, large-scale, and 
numerous-courses designs are needed in the future (Akcayir & Akcayir, 2018; Karabulut-Ilgu et 
al., 2018; Lundin et al., 2018). Chen and colleges (2018) further proposed the critical potential of 
FC in education. It seems appropriate to further understand whether FC approaches work and, if 
so, in what types of situations and contexts. 

Third, other issues might limit background information relevant to the implementation 
and practice of the FC approach. Some examples of this may include the difficulty of the content 
knowledge, instructors' teaching experiences with the pedagogy, or students' self-efficacy of 
using technology tools (Evans et al., 2018). However, the most recent FC studies failed to 
discuss these issues properly or provide sufficient related information. For example, since 
effective pre-class learning in FC has highly relied on students' self-regulation skills, it is an 
essential factor to be explored for the overall success of the FC practice (Cheng et al., 2019). 
Evans et al. (2018) also indicated the need to discuss faculty preparation issues for the FC 
approach. How to provide training to faculty in terms of technology usage, pedagogical design, 
and instructional supports may be important questions to answer as well as other factors in the 
success of FC. 
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The Definition of Learning Effects of FC Needs to Be Clarified 


Results from the FC reviews raised a concern about the definition of learning effects used 
in FC studies to examine the FC effectiveness. For example, Evans et al. (2018) pointed out that 
inconsistency in terminology was a concern in the FC studies. Additionally, when discussing 
possible conceptual ideas related to the effectiveness of FC, most of the FC studies mentioned a 
mix of pedagogical strategies without providing a thorough explanation nor referring to the 
educational or learning theories framework. Blended learning and active learning were two of the 
ideas that were commonly brought up when examining factors associated with the effectiveness 
of FC (Lundin et al., 2018). While blended learning and active learning are considered effective 
strategies to support learning, few studies distinguish the impacts of FC from the effects of 
blended learning or active learning. It aligned with Akcayir and Akcayir (2018)’s argument that 
without identifying the definition of the conceptual terms, it is difficult to determine that the 
shown effect of FC was the result from FC or from other learning modalities. 

In addition to the definition of learning effects, the type of indicators used to examine FC 
effects on learning could also be a potential issue. The retained FC reviews showed that 
academic performance (e.g., test scores and course grades) and self-reported data (e.g., 
participants' motivation and perceptions of the learning experience in FC situations) were the 
primarily common evaluation methods used in FC studies. However, Lundin et al. (2018) argued 
that most of the FC studies in their review chose inappropriate ways to examine the effects on 
student learning. They stated that survey data like pre-and post-test scores or course 
questionnaires could not provide an in-depth analysis of whether or what learning occurred in 
FC. Given that the purpose of FC was to promote higher-level cognitive skills like application, 
creation and behavioral change, lower-levels of assessment like recall in a test may not 
necessarily be considered as an appropriate way to evaluate the effects of the FC approach (Chen 
et al., 2018). To delineate FC's effectiveness, better sensitive indicators to measure the higher- 
level cognitive outcomes need to be developed (Chen et al., 2018). 


Despite the Overall Positive Impact on Learning, the Varied Effects of FC should be 
Further Examined with Sufficient Qualitative and Quantitative Data 


Most of the FC reviews discussing the effect of FC included in this study indicated that 
compared to the in-class or lecture-based approach, the positive impact of FC on students' 
learning outcomes had the effect sizes as small to medium (Chen et al., 2018; Cheng et al., 
2019). Furthermore, lacking reviews that can evaluate the effectiveness of FC systematically 
may result from insufficient data or design information in previous FC studies. Hew and Lo 
(2018) indicated that scant FC studies provided the detailed and necessary information to 
conduct a qualitatively or quantitatively systematic review, which may be why only five FC 
reviews in the present study contained enough statistical data to conduct their studies using a 
software like Comprehensive Meta-Analysis (CMA). Besides the shortage of necessary data for 
statistical investigation from FC studies, the lack of information about the pedagogical design 
and implementation of FC makes it hard to perform qualitative analyses. Cheng et al. (2019) 
indicated that FC researchers seemed interested in reporting the effects of FC on students' 
learning outcomes, rather than examining the learning environment's actual design and 
implementation features (pedagogy). Without the essential information like the instructional 
strategies that applied the FC approach during the online and in-class time or the instructors' 
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teaching experiences related to the FC approach, it would be difficult to know which factors 
contributed to the overall impacts of FC. The above results suggest that more information is 
needed to support the effects of FC on teaching and learning. 

While most FC reviews agreed on the overall positive impact of FC on learning, 
compared to non-FC conditions, in general, the effect size varied. Also, its actual effectiveness 
on learning remains debatable. More detailed qualitative and quantitative data are needed for 
further systematic analysis to provide high-quality evidence to form policy decisions on how best 
to use FC to enhance learning. A more solid theoretical background to distinguish FC from other 
learning approaches like blended learning and self-regulated learning is needed to determine 
what contributes to the learning effect in FC conditions. A stronger focus on the planned studies 
about evaluation or assessment of FC and the development of evaluation tools could provide 
experimental evidence of the impact of FC on teaching and learning. A broader-level study 
design that focuses on longitudinal, large-scale, and multiple courses is needed, as well. 

The authors acknowledge some inherent limitations in this study. In general, the 
discussions in this study are based on the information and findings from the included 13 FC 
reviews. In other words, their study heavily relied on the quality of these review studies. Possible 
problems from previous FC reviews, such as inadequate literature search or improper analysis of 
the findings, can impact the quality and results of this study. Furthermore, although they tried to 
locate high-quality reviews in the first place, it was inevitable to lose detailed information of the 
primary literature. They developed a search strategy to search for FC reviews that were 
published in a major database. However, there is still a possibility that they may have missed 
some relevant FC reviews. Studies that were published after the search date in this review were 
not included either. In addition, the authors did not check if the included reviews selected the 
same FC studies. Therefore, FC reviews that focused on similar research purposes might 
partially consist of the same FC studies. 

As stated by many scholars, the FC approach holds much potential, including more 
hands-on time with students and opportunities for active and collaborative learning. The authors 
are looking forward to seeing more creative adoption of FC as a pedagogical tool that can be 
used with other strategies, rather than substituting or replacing another. They hope to promote 
more effective and engaging practices in teaching that ultimately create a highly positive impact 
on student learning. There needs to be additional research in a broader range of learning 
contexts, while combining FC approaches with other instructional innovations to achieve such 
goals. 
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Abstract 
Recently, the development of problem-solving ability is emphasized. We taught Japanese high 
school students a framework of problem-solving based on Engineering Design. As a result, we 
found that students' awareness of problem-solving ability and self-assessment of problem- 
solving activities were improved. The students are considered to be suitable for the problem- 
solving framework based on Engineering Design. 


Keywords: Problem-solving, Engineering Design, Information 


1. Introduction 


There has been a lot of discussion about the qualities and abilities that should be developed 
for the children of the future. For example, ATC21s has defined 21st century skills. The 21st 
century skills are defined as ten skills in four areas that are considered necessary for living in the 
21st century. One of them is the problem-solving skills (Griffin et al. 2012). In addition, the 
OECD tested problem-solving skills for the first time in PISA 2003. These indicate that attention 
is being paid to the development of problem-solving skills. 

Problem-solving skills have been defined in many ways. The OECD (2012) defines 
problem-solving skills as the skills to understand and cognitively process problem situations 
where the solution is not immediately obvious, as well as a proactive attitude toward the problem. 
Thus, the problem-solving skills are widely considered to be the skills to find an answer to a 
situation where the answer is not yet known. 
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1.1 Problem-solving in Japan 


In Japan, education is standardized throughout the country based on the contents of the 
Course of Study issued by the Ministry of Education, Culture, Sports, Science, and Technology. 
The Course of Study is revised about once every 10 years to improve the content of education. In 
the Courses of Study, problem-solving skills are listed as one of the qualities that form the basis 
of learning. Therefore, it indicates that problem-solving skills should be taught throughout the 
school to improve the quality of educational activities and increase the effectiveness of learning. 

Problem-solving skills are intended to be cultivated throughout school education. In 
school education, information science has been cited as an important subject for developing 
problem-solving skills (MEXT 2016). In Information I, one of the goals stated in the Courses of 
Study is "to acquire knowledge and skills about how to find and solve problems" (MEXT 2018). 
Problem-solving skills have been treated with more emphasis over the years. 

This problem-solving skills has been practiced in various ways in Japanese information 
science. The following are some of the issues that have arisen through the practice in Japan. 
These include teaching knowledge and skills related to problem-solving and the integration of 
lessons on problem-solving skills into the teaching plan. 


1.2 Develop Problem-solving 


To develop problem-solving skills, many studies have been conducted and challenges 
have been identified. First, it has been shown that explicitly presenting and teaching skills is 
effective in developing problem-solving abilities(Mathee and Turpin 2019). In addition, 
Schoenfeld (2013) showed that it is necessary to develop a framework for problem-solving. In 
addition, there is a need to assess problem-solving skills (Griffin et al. 2012). Happner and 
Peterson (1982) developed the PSI to assess individual behavior and attitudes. 

From this, it is clear that there is a need for an explicit framework for problem-solving as 
a skill. Engineering design has shown promise as a framework for problem-solving. 


1.3 Engineering Design 


We focused on engineering design as a problem-solving framework that improves 
problem-solving skills. It is said that there is no unified definition of engineering design (Li et al. 
2016). For example, NGSS (2013) defines three types of behaviors that students who have 
mastered engineering design can perform (A) Define the problem to be solved. (B) Generate 
several solutions. (C) Optimizing the solutions to improve their quality. 

The effects of engineering design on the problem-solving process have been examined in 
previous studies. Problem definition and information retrieval can lead to better problem solving 
(Atman et al. 2007). Although the time spent on problem solving activities is longer, it led to 
better problem-solving activities(Atman and Bursic 1996). However, no conclusion has been 
reached that it improves problem-solving skills (Li et al. 2016). 


2. Purpose 


We suggest that Engineering Design is effective in developing problem-solving skills. 
However, there is a lack of research that examines whether engineering design improves 
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problem-solving skills. The purpose of this study is to investigate the effect of teaching 
Engineering Design to high school students on their problem-solving skills. 


3. Methods 
3.1 Participants 


In July of 2020, this study was conducted in 100 minutes at a private high school in 
Tokyo. The Participants were two classes of 82 third-year high school students, 54 (65.9%) boys 
and 28 (34.1%) girls, who were taking "Information". 


3.2 Procedure 


This practice was carried out according to the procedure shown in Figure 1. First, the 
participants answered a pre-questionnaire survey. The items of the pre-questionnaire were related 
to their perceptions of their problem-solving abilities. Next, the participants worked in groups of 
four or five on problem-solving activities. The problem-solving activity was conducted using a 
chat application on a smartphone. Afterward, the participants self-evaluated themselves and the 
group on the problem-solving activity. Then, we conducted a class to teach engineering design. 
In the class, we explained the framework of the problem-solving framework using PowerPoint 
and handouts. After the class, the students performed the same problem-solving activities as 
before and self-assessed the problem-solving activities in the same way. Finally, a post- 
questionnaire survey was conducted. 

We prepared two types of group work topics. One was a problem-solving activity to think 
about what is needed to make life in a regular classroom more comfortable. I put a limit of 
100,000 yen or less on the budget and asked them to think of something that could be installed in 
about two months. The second was a problem-solving activity in which students were asked to 
think of products that they would like to have in the school store. We put a limit on them to think 
of products within the range of what high school students can afford. 

In teaching the problem-solving framework, the elements of the problem-solving 
framework based on engineering design were taught one by one. In addition, we taught the 
problem-solving framework by having the learners fill in the blanks on the handout that had 
blanks throughout the content. The framework of the problem-solving approach to be taught to 
the learners was developed based on Engineering Design (NGSS 2013, 2017) (Table 1). In 
teaching the problem-solving framework, the NGSS is written in English, it was translated into 
Japanese for teaching. 
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Face sheet - Pre-questionnaire 


Performance Test + Problem-solving activity self-assessment 


Class of problem-solving framework 


Performance Test + Problem-solving activity self-assessment 


Post-questionnaire 


Figure 1. Procedure of research 


Table 1. Framework for problem-solving based on Engineering Design 
Define a simple design problem that reflects the need, with specific success criteria and 
constraints on materials, time, and cost. 
*Understand what the problem is now. 
*Understand what is required to solve the problem. 
*Understand the conditions that must be considered when solving the problem. 


Generate and compare multiple possible solutions to the problem based on how likely 
each solution is to meet the criteria and constraints of the problem. 


Develop *To be able to generate multiple solutions 


*Compare the degree to which each solution satisfies the problem. 


Share ideas to improve the solution. 
Plan and execute unbiased tests that control variables and account for failures in order to 
identify aspects of the prototype that can be improved. 
Optimize Design tests to test solutions. 
*Identify problems with the solution from test results. 
*Collaborate to gather data for evidence. 


3.3 Questionnaire 

To investigate the changes in problem-solving skills, a questionnaire survey was 
conducted. First, a questionnaire survey on self-perception of problem-solving skills was 
conducted before and after the practice. In setting the items for the questionnaire, we selected 22 
items out of a total of 47 items (32 items from the Problem-Solving Inventory (PSI) (Happner 
and Petersen 1982) and 15 items from the scale for measuring attitudes toward problem-solving 
(Emoto et al. 2005 in Japanese)). 

In addition, self-evaluation of problem-solving activities was conducted. The self- 
assessment items included 9 items on the framework of problem-solving and 2 items on the self- 
confidence of the problem-solving activities. The students were asked to self-evaluate the degree 
to which they were able to perform problem-solving activities for their individual problem- 
solving activities and the problem-solving activities of the entire group. 
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In addition, a questionnaire survey was conducted after the practice to investigate the 
usefulness of the framework of the problem-solving approach. Five items were set as survey 
items to investigate the degree of usefulness through the problem-solving activities using the 
framework of the problem-solving approach. 


4. Results 


Based on the collected data, we investigated the impact of teaching the framework of 
problem-solving thinking on the learners. From the results of each questionnaire survey, those 
with missing values were excluded from the analysis, and data from 38 participants (46%) were 
used. 


4.1. Problem-solving skills 


The results of the pre- and post-questionnaires were used to measure changes in the 
perception of problem-solving ability. After processing the reversed items, Shapiro-Wilk's test 
for normality was conducted, but normality could not be confirmed. Therefore, Wilcoxon's 
signed-rank test was conducted on the pre- and post-questionnaires of self-perception of 
problem-solving ability (Table 2). 


Table 2. Self-awareness of problem-solving skills 


Pre Post i ee 


M spd MSD. More 


7 I am able to look at a situation in different 334 105 366 0.71 0.32 1.87 0.35 
ways when solving problems. (R) 


I am able to see things from different 


2: ; 3.55 0.98 3.55 0.80 0.00 — -0.02 0.00 
perspectives. 

3. I try to think about the essence of things 355 0.76 345 0.76 -011 -0.73 0.14 
when I solve problems. 

- Many of the problems I face are too complex 3.03 0.79 3.26 0.95 024 -165 + 027 
for me to solve. (R) 

g vilien Limake:d plan to-solve-a prpblem Lam. ogo p95. SOs 0s Oe: 2092 0.15 
almost certain that I will succeed in doing so. 
When I am faced with a problem, I am not 

® sure that I can handle the situation. (R) pale ee St aw De hee 028 
One of the first things I do when I notice a 

7. problem is to try to find out exactly what the 3.55 0.76 3.45 0.95 -0.11  -0.69 0.12 
problem is. 
When I am faced with a complex problem, I 

8. gather information so that I can define 3.42 0.64 3.61 089 O18 -1.29 0.24 
exactly what the problem is. 

9. After I solve a problem, I do not analyze what 305. 1.09 2.79 102 -0.26 -1.54 0.25 


went well and what did not go well. (R) 
When I am faced with a problem, I come up 
10 with as many ways as possible to handle it 
‘and think about them until I can't think of any 
more ideas. (R) 


2.71 0.90 3.08 0.91 0.37 -2.38 * O41 
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11. 
12. 
13. 
14. 
15. 
16. 
17. 


18. 


19. 


20. 


21. 


22. 


n=38, (R): Reverse score, 5-point Likert scale 


When I decide on an idea or possible solution 
to a problem, I do not take the time to 
consider the likelihood of success of each 
alternative. (R) 

When I am faced with a problem, I stop and 
think before deciding on the next step. (R) 

I generally follow the first good idea that 
comes to mind. (R) 

When choosing a solution, I weigh the 
consequences of each alternative and 3.37 0.82 3.29 0.80  -0.08 
compare them to each other. (R) 

I try to predict the overall outcome of 
implementing a particular solution. 

When I try to think of possible solutions to a 
problem, I do not come up with too many 3.50 0.89 3.39 0.75 -0.11 
alternatives. (R) 

I have a systematic way of comparing 
alternatives and making decisions. (R) 

When confronted with a problem, I usually 
do not consider what external factors are 2.79 0.66 3.18 0.73 0.39 
contributing to the problem. (R) 

When I am confused by a problem, one of the 

first things I do is to investigate the situation 3.11 0.73 3.42 0.76 0.32 
and consider all the relevant information. 

Sometimes I don't stop to deal with my 

problems, I just mess around and move 3.29 0.98 3.29 1.11 0.00 
forward without taking the time to do so. 

When I am working on a problem, sometimes 

I feel like I am groping or wandering and not 3.45 0.83 3.45 0.72 0.00 
getting to the real problem. 

I sometimes make poor decisions and regret 
them later. 


2.84 0.72 3.05 1.01 0.21 


3.63 0.97 3.47 0.95 -0.16 


3.42 1.08 3.18 0.95  -0.24 


3.18 0.77 3.50 0.80 0.32 


2.74 0.69 3.05 0.84 0.32 


4.11 0.89 3.79 0.87 -0.32 


4.2. Problem-solving activities 


-1.28 


-1.13 


-1.15 


-0.41 


-1.40 


-0.65 


-2.00 


-2.70 


-1.80 


-0.04 


-0.07 


-2.13 


* 


7K 


t 


* 


0.24 


0.16 


0.23 


0.10 


0.40 


0.13 


0.41 


0.57 


0.43 


0.00 


0.00 


0.36 


tp<.100, *p<.050, **p<.010 
P 


Based on the results of self-evaluation of problem-solving activities, we measured the 
change in self-evaluation of problem-solving activities. As a result of Shapiro-Wilk's normality 
test, normality could not be confirmed. Therefore, Wilcoxon's signed-rank test was used for the 
pre- and post-questionnaires of self-perception of problem-solving skills (Table 3 for individual 
self-evaluation of problem-solving and Table 4 for group self-evaluation of problem-solving). 


Table 3. Self-assessment of individual problem-solving activities 


Pre Post Mane 3 
aero e A r 
M SD M_ SD Pe 
I. I am satisfied with my problem solving. 3.21 1.06 3.55 1.02 0.34 — -1.84 0.30 
tI was able to define a solvable problem 
2. Ss that meets the needs of the problem 3.08 1.04 3.37 0.96 0.29 -2.07 * 0.34 
oO 


conditions. 
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When defining the problem, I was able to 
3. identify what needs to be solved to be 3.08 0.77 3.26 1.12 0.18 -1.79 fF 0.29 
successful. 
I was able to identify the constraints of the 
situation to be solved when defining it. 


5: I can generate multiple solutions. 2.76 1.20 3.32 0.95 0.55 — -0.83 0.14 


I can compare solutions in terms of success 
and constraints. 

I can improve my solutions by sharing 
ideas. 

I was able to discover improvements to the 
solution. 

I was able to improve the solution based on 
the improvement points 

I was able to plan and execute surveys and 
10. other activities to discover improvements 2.74 0.94 3.03 1.06 0.29 -2.11 * 0.34 
in solutions. 

I think that my problem solving is better 
than others' problem solving. 


n=38, 5-point Likert scale 7p<.100, *p<.050, **p<.010 


3.05 0.83 3.24 0.93 0.18 -2.64 ** 0.43 


2.84 0.84 3.11 088 026 -2.03 * 0.33 


Oa 
dojaaoq 


3.03 1.09 3.37 0.98 0.34  -1.17 0.19 
3.16 1.06 3.32 1.00 0.16  -0.26 0.04 


2.95 0.94 3.21 106 026 -341 +f 0.55 


\o 
oziuda 


11. 2.58 1.14 2.92 1.09 034 -2.33 * 0.38 


Table 4. Self-assessment of the group’s problem-solving activities 


Pre Post Mise 4 
Sf Le. a a NA r 
M SD M SD Bre 
1. We are Sahat wi CUE PLODIee: 361° 090), B76-08F- Oe. 2104 0.20 
solution 
We were able to define a solvable 
2 problem that meets our needs for the 3.47 0.94 3.61 0.87 0.13 -1.54 0.25 
g problem conditions 
Bee A AN ES WOR We DOSE MOONE? 345: 3006361 006: “ORI iD 0.20 
5 to be successful in our definition 
We were able to identify the constraints 
4. of the situation to be solved when we 3.55 0.94 3.58 0.94 0.03 2.45 * 0.40 
defined it 
5. We. yore: able: seencialy, MUlnple oo. id Bow Ihe: VOBE-. °c0.50 0.03 
vy __ solutions 
- : : 
a We were able to compare solutions Mm 318 0994 3.45 0.99 0.26 206 * 033 
= terms of success and constraints 
© We were able to improve our solutions 
‘T : . 3.24 1.01 3.63 0.96 0.39 -1.57 0.25 
by sharing our ideas 
8. We were able to discover improvements 326 1.07 334 1.06 0.08 0.06 0.01 
to the solution. 
i, Fae LS GN AUK 0 MUPIONS eS ROMNON a 0d 380s. Aol 2008 - Siig 0.19 
=. based on the improvements 
E. We were able to plan and carry out 
N Pepe 
10. Suny Gy8) ANG SOME ACUYINGS HAE WOUIE’. Suni, 07-384 Ade. 004 40. 2 ~Oa6 
enable us to discover improvements in 
our solutions. 
ae We oink -Wabeour ‘problem Solving 18: “3.95 igo. B08: 0.844 G24 “201 “e206 
better than others' problem solving 
n=38, 5-point Likert scale *<.050 
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4.3. Usefulness of the framework 


In the post-questionnaire survey, the evaluation of the problem-solving framework was 
conducted. The mean and variance of the results were calculated (Table 5). 


Table 5. Usefulness of the problem-solving framework 


M SD 
1 I was able to understand the framework of problem-solving skills introduced. 3.76 0.85 
2 Iwas able to solve problems using the introduced problem-solving framework. 3.42 0.89 


3 The introduced problem-solving framework fits as a framework for problem solving. 3.61 0.82 


I was able to solve problems easily by applying the framework of problem-solving 


‘ skills introduced. 


5 I was able to solve the problem better the second time than the first time. 3.79 1.12 


n=38, 5-point Likert scale 
5. Discussion 


In this study, we investigated the impact of teaching a problem-solving framework based 
on Engineering Design on the students. From the results of the questionnaire survey, there was a 
change in the self-perception of problem-solving skills. It showed that the students' self- 
perception of their problem-solving skills changed, and they began to think that they were able to 
solve problems systematically after being taught the framework of problem-solving framework. 
In addition, they have come to believe that it is important to come up with multiple ideas for 
solutions when conducting problem-solving activities. They started to think about my own 
situation first when they started problem-solving activities. This was probably because they 
recognized the importance of defining the problem and developing ideas from the framework of 
problem-solving. 

Self-evaluation of problem-solving activities also changed. In particular, the self- 
evaluation of problem-solving behavior related to define increased significantly. However, many 
items did not change in the self-evaluation of problem-solving behavior related to Optimize. 
Insufficient time for problem-solving activities and not being able to optimize problem-solving 
activities may have influenced the results. There was also a difference between the self- 
evaluations of individual and group problem-solving behaviors. This may be because they were 
not able to utilize the problem-solving behaviors that they were able to perform individually in 
their group problem-solving behaviors. 

We found that the learners felt that the problem-solving framework was useful. However, 
they did not give it a high rating, indicating that it needs to be improved. The self-evaluation of 
the Optimize problem-solving activity also did not improve, indicating the need to improve the 
Optimize part in particular. 

Several issues were found in this practice. First, there was no improvement in any of the 
problem-solving activities. In the framework of the current problem-solving approach, it was 
found that the "definition" problem-solving activity had a certain effect. However, the results for 
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"generation" and "optimization" were not sufficient to say that they were effective. Therefore, 
the framework of problem-solving and its teaching methods need to be improved. 

The fact that the individual problem-solving activities could not be applied to the group 
problem-solving activities indicates that there is a need for consideration when conducting group 
problem-solving activities. In this study, we used smartphones, and we felt the need to provide 
support so that discussions using smartphones could be conducted smoothly. 

In addition, there were some problems in the subject matter and time setting of the 
problem-solving activities. In addition, there were some comments from the chat logs that there 
was not enough time for the problem-solving activities, which suggests that there was not 
enough time for the problem-solving activities. In this practice, the time for each problem- 
solving activity was short because we had to conduct two problem-solving activities and a lesson 
on the framework of problem-solving in a short time. To give learners enough time for problem- 
solving activities, it is necessary to clarify the time required for the activities by conducting 
preliminary experiments. 
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Many institutions have adopted best practices in the design of online education programs. However, the literature 
indicates that there is a need for continued research in the area of evaluating and examining online faculty 
satisfaction levels and self-perceived barriers in relation to online education. This study extended research conducted 
by Lloyd et al. (2012) and investigated faculty perceived (interpersonal, institutional, training/technology, and 
cost/benefit) barriers to online education. Statistical analysis revealed three major items: (1) faculty continues to 
experience barriers to online education; (2) faculty is receiving training and guidance in order to become more 
comfortable, and knowledgeable in online settings; and, (3) as faculty online experience increases, the perceived 
barriers decrease. Additionally, faculty provided detailed comments validating ongoing interpersonal, institutional, 
training/technology, and cost/benefit analysis barriers in online education. Higher education’s online programs will 
benefit from intense professional development, training, as well as, open conversations including focus groups, 
composed of administration, faculty, and students. 


Introduction 


According to the Condition of Education 2019 report, (McFarland et al., 2019) online enrollment in higher 
education continues to escalate with 5.5 million students participating in online education in fall 2017. Additionally, 
2.2 million of those students, or 13 percent of the total undergraduate enrollment, were taking courses completely 
online. Between 2017 and 2028, the overall undergraduate enrollment is projected to increase to 17.2 million 
students. Additionally, from fall 1999 to fall 2017, faculty in higher education had an increase of 49 percent, from 
1.0 to 1.5 million. 


The Condition of Education 2019 report is an annual verification, mandated by Congress to summarize the 
latest data on education in the United States. Local, state, and national governments continue to implement a host of 
initiatives, such as, improving student retention and completion rates, cultivating educational programs, increasing 
professional development activities, advancing technology, and development of academic leadership, etc. (Jolley et 
al., 2014). The exponential demand for online options in higher education directly affects faculty. Therefore, this 
current research will focus on understanding faculty perceived barriers to online education in an effort to advance 
research is this area. 


This study expands on the research conducted by Lloyd et al. (2012), which stated that major factors 
affecting faculty’s perceptions of online learning include: (1) interpersonal barriers; (2) institutional barriers; (3) 
training/technology barriers; and (4) cost/benefit analysis barriers. As online learning continues to be demanded by 
students, colleges have constantly increased online offerings due to the advancement of computer technologies. 
Faculty are tasked to accept new challenges via online education. Higher education would benefit from 
understanding major factors affecting faculty perceptions to online learning. A major benefit would include 
innovative guidance, and new opportunites for improvement in online education programs. “The advancement of 
online education has transformed the availability of higher education. Technology and adaptable curricula formats 
have expanded the speed, methods, and approaches in both traditonal and non-traditional education environments” 
(Fogle et al., n.d.), higher education continues to face changes. 


As described by Windes and Lesht (2014), the dramatic changes which have been faced by higher 


education can be explained within the framework of Disruptive Innovation Theory. This theory suggests two types 
of technological change: supporting technological changes and disruptive technological changes. The authors note 
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that a supporting or sustaining innovation serves the needs of existing customers, provides some enhancements; 
however, it does not result in major changes to the market or its audience. In contrast, a disruptive innovation, 
ultimately replaces services or products because they are more accessible to a population of customers, enabling new 
companies to develop and then dominate the industry. Therefore, as emphasized by Windes and Lesht ( 2014): 


Many have pointed to online education as a disruptive innovation within higher education. Students that 
were not well served by traditional colleges have seen their options increase as the number of online 
courses and programs has grown. While initially resistant, many institutions of higher education are now 
attempting to compete with early adopters by developing online programs in order to reach this 
underserved market. (p. 1) 


The Ohio Faculty Council (OFC) which represents all of the faculty at all of the four-year public 
universities in the state of Ohio, advised that higher education in Ohio is at a crossroad, and must make it easier for 
students to attend college and earn quality and affordable degrees. “Today, 45% or more of working age adults in 
other states like Massachusetts, Maryland and Virginia have two- or four-year degrees compared to only 36% of 
working age adults in Ohio” (Ohio Faculty Council [OFC], 2014, para. 2). 


A major strategy for Ohio, as noted by OFC includes a personalized academic program of study, which will 
ideally include the option of courses from Ohio Public institutions delivered in the online learning format. However, 
throughout the literature, it is noted that faculty continue to resist online education. 


Statement of Problem 


Faculty continue to express fears and concern about online education (Davis & Jacobsen, 2014; Eickholt, 
2016; Fogle et al., (n.d.); Gillett-Swan, 2017; Glass, 2017; Reid, 2014), and they are reluctant to teach in the online 
format. Online education which has been recognized as an emergent and profitable field, has developed into a 
permanent fixture in mainstream higher education (Luongo, 2018). Students and employers continue to laud online 
education due to the fact that location and scheduling issues are expunged. Higher education administrators continue 
to examine cost effectiveness and surges in enrollment due to evolving technologies. As reported by Seaman et al. 
2018: 


The proportion of the higher education student body taking advantage of distance education courses has 
increased each of the last four years. It stood at 25.9% in 2012, at 27.1% in 2013, 28.3% in 2014, and 
29.7% in 2015. (p. 11) 


Even after a decade of significant growth in the number of universities with online offerings and students taking 
these courses, the level of uncertainty in relation to online learning amongst faculty continues to remain high (Allen 
& Seaman, 2016). 


Wingo et al. (2017) states that in the United States, faculty are being tasked to teach online at an 
outstanding rate. Faculty are hesitant and reluctant to embrace online teaching due to: (1) interpersonal barriers; (2) 
institutional barriers; (3) faculty training/technology barriers; and (4) cost/benefit analysis barriers (Lloyd et al., 
2012). There are numerous policy items and issues that administrators and faculty need to discuss in relation to 
online environments. Obstacles faced at various stages of online education noted by Berge and Muilenburg (as cited 
in Reid, 2014) include: administrative structure, legal issues, organizational change, technical expertise support, 
student support services, access, threatened by technology, faculty compensation and time, social interaction and 
quality, evaluation/effectiveness (p. 384) 


Faculty are increasingly challenged to rethink their underlying assumptions about teaching and learning online, and 
the roles they take as instructors (Luongo, 2018). Luonogo also reminds us that perceived lack of institutional and 
departmental support is one of the biggest deterrents to teaching online. In the meantime, the competition has 
created other avenues such as edx and coursea. Understanding faculty perceived barriers to online education might 
assist in reducing barriers in this environment. 
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Purpose of the Study 


The purpose of this study is to understand if, and why faculty members have perceived barriers to online 
education. Based on a review of faculty perceived barriers in higher education (Allen & Seaman, 2016; Capra, 2011; 
King & Amold, 2012; Lloyd et al., 2012) more research is needed to comprehend this topic. This descriptive 
quantitative study intended to use the results to assist institutions as they cultivate training programs, and faculty 
recruitment policies for online education, in order to meet the growing demand for this type of instruction. The 
findings of this study contributed to the body of knowledge in the fields of faculty development, online learning, and 
higher education administration. 


Methodology 


Some researchers use the term survey research to denote almost any form of descriptive, quantitative 
research (Leedy & Ormrod, 2013). This current study intended to use survey research to acquire information about 
one or more groups of university faculty and their characteristics, opinions, attitudes and previous experience with 
online education by asking questions and tabulating their answers. This study also intends to investigate the 
relationship between faculty members’ demographic characteristics and interpersonal, institutional, 
training/technology, and cost/benefit analysis barriers. Questions will be answered by the use of an online self-report 
survey instrument and the responses will be analyzed quantitatively. 


Research Questions 


The study attempts to answer the following research questions: 

1. Is there a significant difference in faculty perceived institutional barriers for online education based on full- 
time status? 

2. Is there a significant difference between faculty perceived institutional barriers for online education and 
years of online teaching? 

3. Is there a significant relationship between faculty perceived institutional barriers for online education and 
age? 

4. Is there a significant difference between faculty perceived interpersonal barriers and gender? 

5. Is there a significant difference in faculty perceived technology barriers and previous online courses taken 
related to online teaching? 


Instrument 


The survey instrument proposed as the foundation for this study was created by Lloyd et al. (2012) fora 
study of faculty perceived barriers of online education at North Georgia College & State University. “There are 
advantages to using existing instruments, particularly if they have already been validated and reported to be reliable. 
Reusing an existing survey may also allow for an additional point of reference and comparison” (Eickholt, 2016, p. 
3). The noted survey instrument was “pilot tested twice in order to assess the face validity and clarity of the 
questions (pilot test #1) as well as the ease of use of the web-based survey tool and reporting formats (pilot test #2)” 
(Lloyd et al., 2012, pgs. 3-4). Lloyd et al. (2012) sent three email requests for participation to faculty members, an 
informed consent, and a URL to access the online survey. It was also noted that all procedures were conducted in 
accordance with, and approved by the North Georgia College & State University’s Institutional Review Board. 


This present-day research study will augment the 37-item questionnaire that was constructed, distributed, 
and used in the online survey at North Georgia College & State University by the above noted researchers. This 
researcher received an email confirmation to use the North Georgia College & State University’s online survey (S. 
Lloyd, personal communication, September 1, 2014). 


The instrument for this present study was a self-report questionnaire with four sections. The first section of 
the questionnaire contained seven questions that measured online faculty members experience and perception of 
online education. The second section contained 21 questions that measured faculty perceptions of barriers to online 
education on a Likert scale. The Likert scale asked the participants to rate the extent that they strongly disagree, 
disagree, agree, or strongly agree with the various statements in relation to barriers towards online teaching and 
learning. The third section was an open-ended question, which asked the participant to list their experience with 
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other barriers to online education. Finally, section four was the demographics area, which contained six questions. 


The population for this study included faculty teaching at least one face-to-face or online course during the 
academic year 2019/2020 semester at a four-year degree-granting public Midwestern University. This included 
those teaching undergraduate and graduate degree programs, approximately 800 participants. 


Data collection 


During the academic year 2019/2020, the Midwestern University’s Office of Institutional Research sent the 
recruitment email via the university’s e-mail system (online) to all faculty teaching at least one face-to-face or 
online course and invited them to participate in the study. The email contained a consent letter informing the faculty 
members of their rights as participants. If the faculty member elected to participate, they clicked a link to Qualtrics 
that was available for them to complete the anonymous questionnaire. 


Data analysis 


The statistical tests used included the following: 
One-way ANOVA - this exploration includes only one independent variable with more than two levels. 
Independent samples t-test - Compares two sample means to determine whether the population means are 
significantly different. 
Pearson correlation — an analysis of the linear relationship between two variables, called the Pearson r. 


Results 


During the academic year 2019/2020, the Midwestern University’s Office of Institutional Research sent a 
recruitment email via the university’s email system (online) to faculty teaching at least one face-to-face or online 
course, and invited them to participate in the study. 


Listed are the colleges within the Midwestern University, which were sent an invitation to participate in the 
study: Arts and Letters, Business Innovation, Education, Engineering, Health and Human Services, Law, Natural 
Sciences and Mathematics, Nursing, and Pharmacy and Pharmaceutical Sciences. Of the 792 faculty members who 
were sent an email invitation, 115 faculty members completed the survey from October 29, 2019 until December 4, 
2019. A response rate of (14.5%) was the result. The demographic characteristics of the participants are illustrated 
in Table 1, which include gender, faculty status, academic rank, experience with online education, perceived level of 
comfort and proficiency with technology for online teaching, and years of online teaching. 


Table 1 
Demographic Characteristics of Respondents 
Demographic N % of Sample 
Gender 
Male 39 33.9% 
Female 58 50.4% 
Prefer not to answer 9 7.8% 
Missing 9 7.8% 
Faculty Status 
Full-time 78 67.8% 
Part-time 24 20.9% 
Missing 13 11.3% 
Academic Rank 
Professor 22 19.1% 
Associate Professor 18 15.7% 
Assistant Professor 14 12.2% 
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Demographic N % of Sample 


Lecturer 23 20.0% 
Visiting Instructor 1 0.9% 
Other 28 24.3% 
Missing 9 7.8% 
Experience with Online Education 
No Experience 40 34.8% 
Taught Online Course 69 60.0% 
Missing 6 5.2% 


Perceived Level of Comfort and Proficiency with 
technology for online teaching 


Not Comfortable 13 11.3% 
Sort of Comfortable 47 40.9% 
Very Comfortable 47 40.9% 
Missing 8 7.0% 


Demographic Characteristics of Respondents (continued). 


Years of Online Teaching 


Never 39 33.9% 
1-4 Years 28 24.3% 
5-8 Years 14 12.2% 
8+ Years 24 20.9% 
Missing 10 8.7% 


Research questions. In addition to providing data analysis for the research questions, respondents provided 
rich qualitative data in the open-ended question in the survey. 


Research question #1: Is there a significant difference in faculty perceived institutional barriers for online 
education based on full-time status? As shown in Table 2, an independent samples t-test was used to determine 
whether differences existed in the mean score for perceived institutional barriers based on full-time and part-time 
faculty rank. As shown in Table 2, there were no statistically significant differences in the score (p<.05) for any 
perceived institutional barriers. 


Table 2 
Independent Samples t-test by Institutional Barriers Full/Part-time 
Full-time Part-time 
M SD M SD t(99) p 
Perceived Institutional Barriers 26.8 5.5 26.7 4.3 0.11 >.05 


Comment from a part-time instructor: 
I am a part-time instructor with full-time (50+ hours) administrative job. I teach a mixed class of in-class 
and distance learners. My method of teaching involves several hands-on, team workshops. It is very 
difficult to give the DL’s the same experience with the hands-on exercises. I did try to take a course in on- 
line teaching but it involved more time than I could dedicate due to full-time responsibilities and it seemed 
to start at a level above my starting point. 
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Comment regarding academic support: “If a Chair is not supportive of online course development, they don’t 
encourage their faculty to design courses.” 


Research question #2: Is there a significant difference between faculty perceived institutional barriers for 
online education and years of online teaching? As shown in Table 3, a One-way ANOVA test was used to 
determine the difference amongst the four groups (never taught online, 1-4 years, 5-8 years, and 8+ years). There 
were no statistically significant differences; however, the 8+ years group had a lower mean, as compared to the other 
groups. 


Table 3 
ANOVA Institutional Barriers Based on Years of Teaching Online 


Never taught Taught online Taught online Taught online 


online 1-4 years 5-8 years 8+ years 
F 
M SD M SD M SD M SD (3, 100) p 
Perceived 
Institutional Barriers 28.1 4.9 28.6 4.6 26 3.5 22.7 5.4 8.19 0.001 


One respondent has been teaching since 2002: 
The Online Learning division should be marketing our programs. Many UToledo online programs are 
certified through Quality Matters. UToledo online courses offer direct access to the instructor, usually 
within 24-48 hours, as is not the case with on campus courses. 

One respondent commented on the benefits of online education: 
Online courses help students beyond the expected coursework in that scheduling/planning, time 
management and project management are learned, providing enhancement of skills for undergrads and 
graduate students. 
Online learning provides help lines, tutorials, and a myriad of other support materials for students. This also 
improves student communication skills, both verbal and written. I see no down side to online teaching, 
other than an instructor who does not put in the time and energy to fully transform their on-campus course 
into a quality online course. I have been teaching online since 2002. I have used WebCT and Blackboard, 
but I am also familiar with Canvas, Moodle and many others. 

On the other hand, a respondent with no online teaching experience proclaimed: 
I would like to teach, but getting over the first hump is a little scary. If I had more hands on support from 
other faculty who have taught online or modules that could improve my online skills, I think that would 
help. 

Additionally, some respondents did not have any barriers: None. In fact, I get far more participation in online 

classes than I do face-to-face. 

This respondent favored face-to-face instruction: There is empirical evidence that face-to-face instruction results in 

better learning than the disembodied online experience. 


Research question 3: Is there a significant relationship between faculty perceived institutional barriers for 
online education and age? As shown in Table 4, a Pearson correlation was used to determine whether there was a 
relationship amongst age and perceived institutional barriers. The test showed that a negative relationship existed. 
As age increased, perceived institutional barriers decreased. Or, as age decreased, perceived institutional barriers 


increased. 
Table 4 
Pearson Correlation by Institutional Barriers Based on Age 
Measure 1 2 
1. Age : 
2. Perceived Institutional Barriers -0.15 - 
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Research question 4: Is there a significant difference between faculty perceived interpersonal barriers and gender? 
As shown in Table 5, an independent samples ¢-test was used to determine whether differences existed in the mean 
score for perceived interpersonal barriers based on gender. As shown in Table 5, there were no statistically 
significant differences in the score (p<.05) for any perceived institutional barriers. 


Table 5 
Independent Samples t-test by Interpersonal Barriers and Gender 
Male Female 
M SD M SD (94) p 
Perceived Interpersonal Barriers 142 3.5 14 3.7 0.83 >.05 


Faculty illustrate some of the perceived interpersonal barriers that exist based on gender: 
As a Black female professor, students are less respectful to me than their white and male counterparts. They 
either plead for me to change their grades and if I maintain the integrity of my grading policy, they retaliate 
by writing negative evaluations. They do address me by my first name and or attempt to question the 
integrity of my expertise in the subject. 

This comment discussed difficulties of online environments: 
I am resistant to online teaching because I feel strongly that the most valuable parts of college are meeting 
your classmates and professors, having to show up on time, and the dialogue that happens in class. I 
associate online-heavy curricula with unaccredited, for-profit universities, not with legitimate institutions. I 
enjoy students and want to get to know them which is difficult in an online setting . I also sense that the 
emotional labor component of teaching is extremely limited for online-only faculty. They don't have to 
stand in front of students or develop relationships with students, so they don't get asked for letters of 
recommendations, advice on job interviews, questions, etc. that face-to-face faculty do. My sense is that 
students don't take online courses as seriously as their face-to-face classes. Students will freely admit even 
to faculty that they take online classes because they are "easy" or because they can use the book during 
tests. That is highly problematic. 

This respondent commented on instructor creativity, morale, and motivation: 
I think that some of the barriers listed about being impersonal can be true, but it is up to the instructor to 
ensure that the online class is finding ways to make it more personable. However, this also makes it a much 
more time consuming class to teach. It is definitely harder to engage students who are only taking an online 
class so they don't have to "show up" to a physical class. That mindset can be a barrier for instructors. 
Adjunct instructors might also feel less connected to the department and other instructors if they are not 
physically present. Having a connection with other faculty in the department can help with creativity, 
morale, and motivation. 


Research question 5: Is there a significant difference in faculty perceived technology barriers and previous 
online courses taken related to online teaching? As shown in Table 6, an independent samples f-test was used to 
determine whether differences existed in the mean score for perceived technology barriers based on if the 
respondent has taken previous online courses in relation to online teaching (yes or no). As shown in Table 5, there 
were no statistically significant differences in the score (p<.05) for any perceived institutional barriers. 


Table 6 
Independent Samples t-test by Technology Barriers and Previous Courses 
Yes No 
M SD M SD___t(103) p 
Perceived Technology Barriers 4.1 1.4 4.3 1.2 -0.69 >.05 


This respondent discussed technology barriers: 
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It sometimes feels more difficult to create meaningful, active discussions or activities to engage students. 
There are always options to use discussion boards, and have students create short videos, however, it feels 
to me there is always a challenge in engaging students online, when compared to in-classroom. 


Comments below state optional forms of technology: 
If you have a good platform like Zoom where you can see each other and divide into virtual small 
discussion groups, share your screen, and do the class in real time it is a really great way to teach and learn. 
It's more inclusive for distance learners. I do think that there is something different that happens in a face to 
face situation that is in person that is different from the online. I am not sure if it's different enough that the 
lack of face to face is a barrier. It might be for better to have blended classes some online class times and 
some in person especially for health and human services were dealing with people in person is part of the 
profession. 

Other technology concerns: Poor support of computer hardware and software from university needed for effective 

course development and monitoring and availability of training to use online system. 


Conclusions 


The purpose of this study was to understand if, and why faculty members have perceived barriers to online 
education. Results indicated that faculty continues to experience various barriers online. However, faculty report 
being comfortable in the online environment, and continue to embrace this form of educational delivery. 


Faculty in this study, based on the specific research questions, do not have perceived institutional barriers 
based on full-time status. Results also revealed that as faculty online experience increases, faculty perceived 
institutional barriers decrease. Additionally, there are no perceived interpersonal barriers based on gender; and, no 
perceived technology barriers based on previous online classes taken in relation to online teaching. There is a 
relationship amongst age, and perceived institutional barriers; as age increased, perceived institutional barriers 
decreased. 


On the other hand, faculty comments provided rich, practical, and extensive documentation of ongoing 
interpersonal, institutional, training/technology, and cost/benefit analysis barriers. The study also highlighted 
additional critical barriers experienced by faculty, which included the lack of student preparation in the online 
environment, issues with learning management systems, and questionable support of academic leaders towards 
online education. The online environment has become even more integral to scholarship based on the pandemic 
health concerns throughout the world. 


Online education is no longer, just a preference. Administrators, faculty, and students are on notice as to the 
urgency, and significance of this form of educational delivery. Planning must also include transformation of face- 
to-face courses to online delivery. Additionally, faculty will benefit from intense professional development, training, 
as well as, open conversations including focus groups, composed of administration, faculty, and students. Online 
education continues to evolve, and higher education continues to advance, and embrace this challenge. 


In summary, this research study sought to understand why faculty continue to express fears and concerns in online 
environments. The data provided generous examples of current interpersonal, institutional, training/technology, and 
cost/benefit analysis barriers. The significance of the study provides administrators with guidance, and opportunites 
to improve current practices in the development of online education. Guidance includes understanding the 
technology acceptance model, online education, pedagogy, and tackling the critical barriers from question #29. The 
domain of education has changed forever; and, it continues to face new challenges worldwide. This research 
provides evidence that it is imperative to address faculty perceived barriers in the “everchanging digital world” 
called online education. 
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TraceMe + Pedagogical Agent = Morgan: Personalized Learning With AI 


Fatimah Wirth, Ph.D. 


Abstract 


Online courses are often structured using a cookie cutter approach and even when they 
are personalized, they are done based on instructor/instructional designer experience. In this 
research study, the learners' paths was traced throughout the online course. Georgia Tech 
Professional Education (GTPE) collaborated with Professor Alain Mille from the French 
National Center for Scientific Research and France's Universite Numerique and Pierre-Antoine 
Champin from Claude Bernard Lyon 1 University in France and the LIRIS research center and 
used their proprietary software called TraceMe to collect data in a Georgia Tech Professional 
Education (GTPE) course. The research question addressed was how learners navigate through 
an online course. The findings from this research study will help identify trends/paths that can be 
used to restructure curriculum and presentation of content. Understanding the learner's 
navigational and interactivity patterns will inform effective course design (Stein, 2014). The 
findings will also help create an artificial intelligent pedagogical agent that provides a 
personalized path. The AI pedagogical agent will pop up at the point where the learner is 
frustrated or stuck. 


Introduction 


The Covid-19 pandemic has forced everyone to learn in new ways. It has ripped us from 
the traditional classroom environment and forced us to think of education and learning in new 
and novel ways which may have been unthinkable pre-Covid-19. 

Although this research study predates the Covid-19 pandemic, it even more relevant now. 
Once the research data had been analyzed, the Georgia Tech Professional Education (GTPE) 
researchers will be able to determine how the learners navigate through a course based on the 
trace data collected. This will help identify trends/paths that can be used to restructure 
curriculum and presentation of content. Understanding the learner's navigational and interactivity 
patterns will inform effective course design (Stein, 2014). Georgia Tech Professional Education 
collaborated with Professor Alain Mille from the French National Center for Scientific Research 
and France's Universite Numerique and Pierre-Antoine Champin from Claude Bernard Lyon 1 
University in France and the LIRIS research center and used their proprietary software called 
TraceMe to collect data in a GTPE course. 

The aim of this research project was to trace the path that a learner takes during an online 
course session using javascript applied to each course page within the Learning Management 
System. The path that is traced includes how the student peruses the course menu and the content 
within a course. The data collected from the trace of the learner's path through the online course 
can be beneficial in that the researchers will be able to find out where learners have issues within 
the course. The issues may be due to navigation, instructions or course content. As a result, the 
researchers will be able to discover trends or themes when the data is coded. These trends or 
themes will lead to better online course structure, course content and course navigation. This will 
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then improve the quality of the online courses so that participants get a more positive and 
beneficial online course experience. 


Research Questions 


The aim of this research project is to trace the path that a learner takes during an online 
course session using javascript applied to each course page within the Moodle Learning 
Management System (LMS). The path that is traced includes how the student peruses the course 
menu and the content within a course. This will allow us to identify areas within the course that 
need improvement. 

The research questions investigated in this research study were as follows: 

e Identify where and when learner engagement drops off 
e Identify where content needs to be clearer or re-emphasized 
An IRB protocol was submitted and approved before research and data collection started. 


Literature Review 


New challenges in information technology has forced institutions of higher learning to 
rethink education. Many institutions have turned to Artificial Intelligence (AI) to improve 
education. According to Yolvi, et al.: 

The formats based on artificial intelligence promise a very substantial improvement in 

education for all the different levels, with an unprecedented qualitative improvement: 

providing the student with an accurate personalization of their learning tailored to their 
requirements, managing to integrate the various forms of interaction human and 
information and communication technologies. 

(Yolvi, et al., 2019, p.536) 

Artificial Intelligence creates computing systems that have the ability to learn from 
information in its environment and provide adaptive behaviors to assist learning. For the past 30 
years, many institutions of highly learning have focused on creating AI based systems to mimic 
human tutoring. These systems are highly autonomous, interactive and adaptable (Qin, et al., 
2020). 

AI can also assist those with intellectual disabilities. These learners need a wide range of 
learning needs and do not learn in a linear or hierarchical way. The use of artificial intelligence 
tools for education can help move a student from a negative state to a positive state of learning. 
Students who are frustrated or stuck at a certain place in a course can be nudged to move into a 
more positive state of engagement. (Standen, et al., 2020) 

However, so far, even though there may be intelligent tutors powered by AI, there is yet 
not a single one that provides just-in-time assistance where the AI pedagogical agent actually 
pops up at the time of frustration or being stuck in one place in a course and lets the learner know 
that there are other pathways that they can follow in order to get unstuck. This research study 
aims to do just that. 


Methodology 


The instructional designer reached out to at least three instructors teaching Georgia Tech 
Professional Education online courses in Fall 2017. An instructor was chosen based on the 
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interest of the professor to participate and the ease of navigation of the course. The duration of 
the subject participation would be the duration of the course. 

A week before the start of the course, the students in the course were sent a Qualtrics 
survey asking for voluntary participation in the research study. The students had to enter their 
first name, their last name and their email addresses. A consent form was attached to the email. 
In addition, before taking the background or end of course survey, the participants in the research 
study were provided with verbiage that said that if they took the survey, then they were 
consenting to taking part in the research. 

The students first name, last name and their email address were asked so that the sample 
population would be identified. Once this population has been identified, their actual trace data 
will be anonymous. Their names will not be attached to the trace data in any way. The system 
will generate a random ID for each student based on CAS authentication. So each time the 
student logs into the course, the same random ID will be used for the student. There will not be a 
key or any document that will link the names to the random ID generated at any point during the 
study. 

A new role called "Non Participant" was created for all those who did not fill out the 
survey or who did not want to participate in the research study. The new role was manually 
added to each student who did not want to participate. By adding this role, these students would 
not be tracked. These students would therefore have two roles, namely the Student role and the 
Non Participant role. This exclusion from participation in the study was created via the use of an 
HTML block in the Moodle Learning Management System used by Georgia Tech Professional 
Education. The HTML block would be called KTBS. The Non Participant role will ensure that 
the students who do not wish to participate will not see the KTBS HTML block when they 
logged into their course. This would mean that they will not participate in the study. The students 
who agreed to participate in the research study will see the KTBS HTML block when they log 
into their course. In addition, they will see a large red dot on the top right of the page. 

When the students who had agreed to participate in the research study logged into the 
course, their every action in the course would be traced. This was an automatic process and the 
students would not have to do anything to initiate the process. If the students who had agreed to 
take part in the study did not want their actions traced, they can click on the red dot. The green 
dot will then become a red dot. This will mean that their actions in the course are not being 
tracked. If they log out of the course and then log back in, then the green dot will automatically 
appear again. This means that they will have to uncheck the check box again if they do not want 
to be tracked. The tracing of the data will not impact or change the normal user navigation or 
course setup in terms of activities and assessments. 

The students were informed that there were no known risks for participating in the study. 
All student information would be kept confidential. All records would be kept under a code 
number and not by student name. However, students were warned that data may exist on backups 
or server logs after the end of the research project. All data would be deleted three years after the 
completion of the study. 

The students were also informed that the trace data would be on the Amazon Cloud 
server. The data collected would be anonymous and only the researchers would have access to 
this server. Only the researchers would be able to download the data as a CSV file. 

The students who agreed to participate in the course were identified through the response 
to the survey and given access to a consent form. In the consent form, students were notified that 
the software called “TraceMe” would be activated when they logged into the course. They would 
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see a red dot next to their names on the top right and that signified that their actions in the course 
was being traced (see image below). 


*Fatimah (4) Logout 


These students were also informed that if they wished not to be traced on a certain day or 
if they no longer wished to have their actions in the course traced, then they could click on the 
red dot next to their names on the top right. The red dot would then turn into a grey square and 
this would denote that they were no longer being traced (see image below). 


*Fatimah (4) Logout 


Grey Square 


The students who did not wish to participate would not have access to the software and hence 
would not see a red dot next to their names on the top right. They would participate in the course 
as any other student in an online course would. 

In addition, the participants will have to fill out a background survey and an end of 
course survey. The background survey will collect demographic information of the participants. 
The end of course survey will ask questions about the course design and navigation. 

The background data will be collected once at the beginning of the course through the 
background survey. The end of course survey data will be collected once at the end of the course 
through the end of course survey. The end of course survey will determine the difficulty level of 
the content and the navigational ease of the course. The data collected from the background and 
end of course surveys will be correlated to the trace data. For example, the trace data and the 
background data can be correlated to show how younger participants navigate versus how older 
participants navigate through the course. There may also be other correlations in terms of 
navigation through the course such as gender, educational background, reason for taking the 
course, etc. The trace will be continuously sending data on student navigation of the online 
course to the server. This data will be captured for the entire duration of the course. The data 
collected will assist the Georgia Tech Professional Education instructional designers to figure out 
where students get stuck within a course and what can be done to alleviate that. There could be 
follow up interviews with each student who had difficulties to find out why they got stuck and 
what would have made either the navigation or the content better meet their needs. 


Course Design 
The Learning Management System at Georgia Tech Professional Education at the time of 
the research was Moodle hosted by Moodlerooms. The course chosen for the research project 


was VET": Military Transitions to the Workforce. The course was chosen based on the fact that 
it had a clean interface and was easy to navigate. The course was also designed based on the 
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Standards from the Quality Matters Higher Education Rubric, Sixth Edition and had enough 
activities to make the course engaging. 

The tracing software that we piloted was developed by Dr. Alain Mille from Liris (Trace- 
Based System - a project of the TWEAK Team) and Pierre-Antoine Champin from Claude 
Bernard Lyon | University in France. The trace engine was installed on a server and javascript 
code was appended to each course page in Moodle. The trace would be automatically activated 
as soon as the learners who had agreed to participate in the research study logged into their 
course. The tracing software then would record and archive all paths taken by the learners. The 
tracing of the data will not impact or change the normal user navigation or course setup in terms 
of activities and assessments. 

The data collected from the trace of the learner's path through the online course can be 
beneficial in that the researchers will be able to find out where learners have issues within the 
course. The issues may be due to navigation, instructions or course content. As a result, the 
researchers will be able to discover trends or themes when the data is coded. These trends or 
themes will lead to better online course structure, course content and course navigation. This will 
then improve the quality of the online courses so that participants get a more positive and 
beneficial online course experience. 


Data Collection 


Data was collected from the background survey, the end of course survey and the trace 
from the TraceMe software. Data from the background survey and the end of course survey will 
be analyzed quantitatively and/or qualitatively. These data may also be exported to qualitative 
and/or quantitative software packages. 

Dr. Champin created a Pilot HTML block in the course, to host the script that would 
collect the data when the student who had agreed to participate in the research project logged 
into the course. The student would see the red dot on the top right of the page when data was 
being collected. 

The trace data collected from the Amazon Cloud server would be analyzed qualitatively 
and/or quantitatively. The data is anonymous and only the researchers have access to this server. 
In addition, the trace data would be exported to qualitative and/or quantitative software 
packages. Open coding would be used to discover trends, themes or confounding variables 
within the qualitative data collected. The data may also be cross-tabbed across multiple data 
sources (e.g. background survey, end of course survey and trace data from plugin) in order to 
discover trends, themes or confounding variables. The tracing of the data will not impact or 
change the normal user navigation or course setup in terms of activities and assessments. 

The data collected from the participants of the research study will be kept confidential to 
the extent allowed by law. Participant records will be kept under a code number and not by 
name. Participant records will be kept in a locked file if printed and on a secure desktop or laptop 
if electronic. The data on the desktop or laptop will be kept secure by locking the desktop or 
laptop screen with a user login and password. When the desktop or laptop computer is not in use 
the screen will be locked. Only the researchers will have access to these files - printed and/or 
electronic. Participants in the research study will only be identified by number. Data correlation 
will use this number to match data from various sources to the participant. Researchers will share 
data and collaborate on a private Office 365 site. The site will only be available to the 
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researchers. All data (printed and electronic) will be destroyed three years after the completion of 
the study. 


Data Analysis 


Data collection took place in Spring of 2018. The chosen course, VET”: Military 
Transitions to the Workforce, went live on February 5, 2018. There were 26 students in the 
chosen course and all 26 students agreed to take part in the research study. The students were 
given 6 weeks to complete the course which had 4 modules, 4 end of module quizzes, 5 
assignments and 7 videos to watch. Data was collected on an AWS server. The Amazon Cloud 
external server only collected anonymous data. The researchers had access to this server and 
were able to download this data as a CSV file. 

In just 24 days, 2371 pieces of data was collected. The data collected included visiting a 
page, submitting an assignment, clicking a link, clicking a button, changing page view. 


Figure 1 
Preliminary data obtained from VET’: Military Transitions to the Workforce course. 
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Of more interest however, was how the students were making their way through the 
course and where they got stuck or spent the most amount of time. However, data collection and 
data analysis had to stop before the GDPR rules went into effect on May 28, 2018 as the 
researcher collecting the data was Pierre Anton Champin who resided in France. Pierre handed 
over the data to the researcher in the United States and data analysis has stalled since then 
because of the new GDPR rules. 


Work in Progress 


After data collection ceased due to GDPR restrictions, data analysis also ceased. This is 
due to the fact that the researchers who would be analyzing the data were from France. Currently 
there are more than 40,000 bits of data waiting to be analyzed. 

The aim of the project has also changed from identifying two research problems to the 
creation of an AI pedagogical agent that would address the two research problems. The first 
research problem was to identify where and when the learner engagement drops off. The second 
research problem was to identify where content needs to be clearer or reemphasized. The 
pedagogical agent, named Morgan, would pop up at the point when engagement drops off. The 
pedagogical agent will provide either encouragement, explanation or a new path for the learner 
to follow so that the learner can complete the task. In addition, research partners will need to be 
identified to assist in the process of the AI creation based on the current data collected. 

However, since Georgia Tech Professional Education now uses Canvas as the Learning 
Management platform instead of Moodle, replication of research needs to be conducted in the 
same course in Canvas. 
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Abstract 


Students’ satisfaction and perceived learning outcomes are critical determinants of the 
effectiveness of a learning system and provide the needed competitive edge at the end of the 
learning experience. While studies are bound that examined these factors within e-Learning 
systems in disparate contexts, there are few literatures that investigated the variability within the 
population based on the e-Learning experiences and the employment status of the students. 
Therefore, in this study, we developed an integrated model and utilized it to investigate the critical 
predictive factors (CPFs) that promote students’ satisfaction and perceived learning outcome 
within an e-Learning environment in a developing context, specifically Nigeria. In addition, the 
study examined the moderating effects of e-Learning experience and employment status on 
students’ satisfaction and perceived learning within the e-Learning system. 267 responses were 
received from a Google form shared link together with a physically administered questionnaire 
survey instrument and was analysed using the Partial Least Squares Structural Equation Modelling 
(PLS-SEM) multigroup analysis technique (MGA). From the 21 Paths evaluated, results show that 
e-Learning experience and the employment status of the students moderated 11 relationships 
within the model, thus, confirming the relevant hypotheses, while 10 hypotheses were not 
supported. The discernment made and the implication for theory and practice for the institutional 
management of the ODL system in Nigeria were discussed. 


Keywords: e-Learning, Distance Learning, Digital Technologies, Internet, Satisfaction, 
Perceived Learning 


Introduction 


Technology and the internet have become intricately woven and intertwined into the fabrics 
of society, in particular, institutional managers within the educational sector have come to terms 
with the realities of technology as a major driver, catalyser, an accelerator and a multiplier of the 
opportunities within the different facets of the sector. However, these affordances are made 
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possible on account of the growth of the internet and internet technologies. Recognising these 
immense factors, institutions across the globe are investing in these technologies to multiply and 
scale-up access and provide opportunities for more citizens to advance their education, hone their 
skills, and contribute to national economic growth and development (Mtebe, 2020). In addition, 
the deployment of technology enhances the institutional reputation and increases its coverage, 
consequently attracting research interests and advancing knowledge and academic development. 


These transformations in the distance education spectrum have evolved from paper based 
correspondence education to modern day electronic Learning (e-Learning) model(Dick, Akbulut, 
& Matta, 2020). Consequently, disparate pedagogical approaches have emerged, terms such as 
online learning, hybrid learning. blended learning, virtual learning, e-Learning have continued to 
dominate the education space. Recently, the Corona virus pandemic (COVID-19) challenge 
birthed the emergency remote learning model leading to the escalation in the application of e- 
Learning resources for learning (Aboagye et al., 2020).One common feature of the digital learning 
models is the centrality of digital technologies as the delivery medium and the eclipse of time, 
space, and location inhibitions. A situation Moore, (2013) described as the transactional distance 
which is characterized by the dialogue that happen between the learner and the instructor, or 
facilitator. A scenario that has significantly shifted the burden of learning on to the students. 
Researchers such as Moore, Dickson-Deane, & Galyen,( 2011) have pointed out the 
inconsistencies in the conceptualizations of e-Learning. 


These researchers argued that stake holders tend to conflate the diverse models in their 
definitions. In that respect, Tulinayo, Ssentume, and Najjuma, (2018) defined Digital Technologies 
as a broad range of tools, services and applications in the form of software and hardware that are 
used to facilitate services and activities through electronic medium. They are used for creating, 
storing, processing transmitting and displaying information. When deployed for 
educational/pedagogical purposes they are often referred to as e-Learning resources. Over the 
years, researchers have variously defined the concepts of e-Learning. For instance, Stefanovic, 
Nikolic, Drapsin et al., (2011) described e-Learning as learning via internet enabled by digital 
technology and it is a major phenomenon adopted by institutions to improve the teaching and 
learning process. In alignment, Hussein, Daoud, Alrabaiah, and Badawi, 2020); Binyamin, Rutter, 
and Smith, (2020) describes e-Learning as an all-encompassing concept that includes all forms of 
teaching and learning that occur partially or completely using digital technologies. In this study, 
we considered the Open and Distance Learning process at the National Open University of Nigeria 
(NOUN) as an e-Learning given that (90% of the delivery process is conducted online within the 
integrated learning management system called NOUNiLearn. (www.nounonline.net). Also, 
because e-Learning success can be measured through multiple perspective and tools (Serdyukov, 
2020; Stefanovic et al., 2011;Authors, 2020), we considered students’ satisfaction and perception 
of learning as appropriate measures of the NOUNiLearn system effectiveness. Customer 
satisfaction is a widely accepted measure of the acceptance of a product in business research and 
economics. In e-Learning and education by extension, it may be the reflection of the success of 
the learning experience. Hence, this study considers students' satisfaction and the perceptive 
fulfilment of their learning expectations as a plausible measure of the effectiveness of 
NOUNiLearn. 


Many factors have been advanced as critical predictive factors of students' satisfaction with 
the e-Learning system; student factors, instructor factors, interaction factors, motivational factors, 
quality factors, learning environment, as well as the resources deployed for the learning experience 
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play important roles in students’ satisfaction within the e-Learning system (Yunusa & Umar, 2021; 
Alhabeeb & Rowley, 2018)). Therefore, these factors must be considered to ensure successful 
learning process within the system (Stefanovic et al., 2011). The National Open University of 
Nigeria (NOUN) was established in the mode of the Open University UK, Allamal Iqbal Open 
University Pakistan and many others across the world. However, there is limited empirical 
evidence on students' perspectives of the effectiveness of the system measured through the lens of 
students' satisfaction and perceived learning based on their learning experiences. Therefore, this 
study is pertinent, moreover, as Serdyukov, (2020) asserted, though, e-Learning offers the benefits 
in terms of access, inclusiveness, and multiplying the spaces for learning, It also harbours some 
challenges to the students that affects them in different ways that leads to their withdrawal, dropout 
and poor learning motivation and outcomes (Carr, 2000 as cited in (Aboagye, Yawson, & Appiah, 
2020). Serdyukov argues that the e-Learning environment tend to exert some influence on the 
learners cognitively, behaviourally, and learning expectations. Nonetheless, the need for career 
advancement, improved skills, and certification among others, has increased the support for online 
learning among adult citizens in different contexts (Bolliger & Halupa, 2018; Bolliger, & 
Martindale, 2004; Allen et al., 2016). Therefore, we considered it plausible to measure the 
effectiveness of the e-Learning platform from the perspectives of the learners’ satisfaction with 
their learning experiences. The study is significant as it provides students’ perspectives of the 
system that might help to improve critical sectors of the system, provide additional empirical 
grounding on the effectiveness of the e-Learning system in Nigeria and inform effective 
curriculum design and development within the e-Learning system in Nigerian context. 


Moderating Effects: 
*e-Learning Experience 
* Employment Status 
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Figure 1. Conceptual Framework of the study 
Hypotheses development 


Based on review of literature, an integrated framework was conceived, and it comprised 
of the critical predictive factors (CPFs) of satisfaction and perceived learning (Yunusa & Umar, 
2020). As conceptualized in Figure 1. The framework was developed from the quality factors of 
the DeLone & McLean Information System Success model (D&MISS) (DeLone & McLean, 
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2003), Theory of Transactional Distance (TTD) (M. G. Moore, 2013), Self-Determination Theory 
(SDT) (Deci & Ryan, 2015) and the User Characteristics factors (Ozkan & Koseler, 2009; Sun, 
Tsai, Finger, Chen, & Yeh, 2008) ; Sun et al., 2008). The framework was formed because a single 
model may not be sufficient to explain the phenomenon under investigation. Hence, since a 
nomological framework might lead us to the desired study outcome, it is considered plausible. 
Consequently, demographic variables (e-Learning experience and Employment Status were added 
to the fray since they improve the explanatory power within the model based on the 
recommendation by Venkatesh, Morris, & Davis, (2003).We draw on previous studies that adopted 
the use of integrated models to answer to pertinent e-learning questions and provide better 
understanding of various concepts such as behavioural intention and continuance intention to use 
of MOOCs (Zhou, 2016), examination of relationships among students self-determination, 
technology acceptance, and satisfaction to use MOOCs (Joo, So, & Kim, 2018), willingness to 
develop sport tourism (Lin, Chen, Tan, Lee, & Yang, 2018), university students level of 
satisfaction and readiness for e-learning (Topal, 2016), as well as measuring technology 
acceptance level of teachers (Koral Giimiisoglu, 2017), — studies that were predominantly in the 
context of developed countries. 


The following hypotheses were formulated based on the research objectives. 
E-learning Experience Moderating Effects 


Researchers; Lu & Chiou, (2010); Yukselturk, (2009) reported conflicting outcomes in 
their study. While Lu & Chiou (2010) identified job status as a moderating factor in the linkage 
between system quality and satisfaction, Yukselturk, (2009) could not establish any significant 
moderating effect of age, gender, educational level and previous online experience on the students’ 
satisfaction with online learning experience. Consequently, more research is required to better 
understand these phenomena, though, there are argument regarding the justification for including 
moderation analysis in model formation (Holland, Shore, & Cortina, 2017) . Nonetheless, we 
hypothesize that: 


H1: (a, b, c, d, e, f, g, h, i, j): e-Learning experience moderates the effects of (CQ, SQ, SVQ, 


cMOT, aMOT, IF, SF, SSI, SII, SCI) on students’ satisfaction (SAT) within the e-Learning 
environment. 


H?: (a, b, c, d, e, f, g, h, i, j): e-Learning experience moderates the effects of (CQ, SQ, SVQ, 
cMOT, aMOT, IF, SF, SSI, SI, SCI) on students’ perceived learning (PLN) within the e-Learning 
environment. 


H3: e-Learning experience moderates the effect of students’ satisfaction (SAT) with the e- 
Learning system on perceived learning outcome. 


Employment Status Moderating Effects 


Over the years, researchers have investigated the effects of demographic factors on e- 
Learning effectiveness in higher education at different contexts (Islam, Abdul Rahim, Liang, & 
Momtaz, 2011;).(Tarhini, Hone, & Liu, 2014b; Wang et al., 2009) results show that conflicts exist 
regarding the moderating effect factors within the system thus, warranting further research in that 
regard. Islam et al., 2011 reported significant moderating effects of age, program of study, and 
level of education on e-Learning effectiveness. Against this backdrop, we hypothesize that: 


H4: (a, b, c, d, e, f, g, h, i, j): Employment Status moderates the effects of (CQ, SQ, SVQ, cMOT, 
aMOT, IF, SF, SSI, SII, SCD) on students’ satisfaction within the e-Learning environment. 
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H5: (a, b, c, d, e, f, g, h, i, j): Employment Status moderates the effects of (CQ, SQ, SVQ, cMOT, 
aMOT, IF, SF, SSI, SII, SCD) on students’ satisfaction within the e-Learning environment. 


H6: e-Learning experience moderates the effect of students’ satisfaction (SAT) with the e- 
Learning system on perceived learning outcome. 


Purpose and Method 
The study sought to answer the following research questions: 


(1) What are the critical predictive factors of e-Learning satisfaction and perceived 
learning within the e-Learning environment? 

(ii) | How do e-Learning experience and Employment Status moderate students’ 
satisfaction and perceived learning within the e-Learning environment? 


The study adopted the quantitative research design approach (Clark & Creswell, 2015) A 
Google form online link was shared on the WhatsApp platforms of the target respondents and the 
questionnaire was physically administered to gather the data. The respondents are Bachelor of 
Education undergraduate students who had enrolled in the ODL e-Learning programme for at least 
four semesters. This category of students was targeted because they have used and garnered 
enough experience to report on their satisfaction and learning expectations of the system. 
Researchers have argued that students are better placed to define the effectiveness of the e- 
Learning environment because they are the target clientele and the most users of the system (Sher, 
2009). The survey instrument included questions with items drawn from previous research 
instruments with established validity and internal consistency reliability. And based on quality 
factors, interaction, motivation as well as user characteristic factor items. The students are 
expected to respond to the five-point Likert scale statement (ranging from 1I-Strongly Disagree 
(SD) to Agree 5-Strongly (SA) to reflect their perceptions with the e-Learning system predicated 
upon their prior learning experiences. 


Findings and Discussion 


Responses were gathered from 285 students whose age range between 18 to 55 years, 64% 
(n-171) are male and 36% (n-96 are female). Also,64.4%(n-172) are employed while 35.6% (n-95) 
are not-employed. Meanwhile, 85% (n-228) of the respondents have significant e-Learning 
experience. Consequently, 267 responses were found useable for analysis using the Partial Least 
Squares Structural Equation Modelling (PLS-SEM). PLS-SEM was chosen because it is widely 
accepted as a useful tool for investigating causal model relationships (regressions) and moderating 
effects. PLS-SEM is preferred for its relative advantages over first generation statistical analysis 
techniques especially been amenable to all kinds of data (normal and non-normal) (Ghasemy, 
Teeroovengadum, Becker, & Ringle, 2020; Sarstedt & Ringle, 2020). PLS-SEM can 
simultaneously test the plausibility of an entire collection of propositions of a causal theory, can 
model multiple independent variables (IVs) and multiple dependent variables (DVs)(Lowry & 
Gaskin, 2014). 


The analysis was conducted using the SmartPLS version 3.2.8 (Ringle, Wende and Becker, 
2015) after data cleaning and ensuring that basic statistical assumptions were met. As noted by 
Trochim, (2006) data analysis encompasses three fundamental steps viz: Data preparation, data 
description and Hypotheses testing. Thus, based on the recommendations by Hair, Sarstedt, & 
Ringle, (2019); Sarstedt, Ringle, & Hair, (2017) a two-step approach was followed to determine 
the convergent validity and the composite reliability indices as quality criteria, while the 
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discriminant validity and collinearity issues were also assessed followed by the structural model 
analysis to test for relationships. The results for the path coefficients and differences in terms of 
Employment status and e-Learning experience between the groups is presented in Tables | and 2." 


Table 1: Significant values for the moderating effects of e-Learning experience 


t-value 

Relationship ; Beta value More : More p-value More experience vs 

experience -Less experience experience -Less Less experience 

experience 

CQ >PLGN 0.697 3.053 0.001 
CQ > SAT -0.439 3.469 0.000 
IF > PLGN -0.284 0.970 0.167 
IF >SAT 0.932 6.684 0.000 
SAT > PLGN -0.204 0.880 0.190 
SCI > PLGN 0.399 2.374 0.010 
SCI >SAT 0.037 0.264 0.396 
SF >PLGN 0.735 4.168 0.000 
SF >SAT -0.334 2.673 0.004 
SII > PLGN 0.275 1.418 0.079 
SIL > SAT -0.286 2.715 0.004 
SQ >PLGN -0.563 3.241 0.001 
SQ > SAT 0.366 3.492 0.000 
SSI > PLGN -0.269 1.223 0.112 
SSI > SAT -0.268 2.257 0.013 
SVQ > PLGN -0.234 0.986 0.163 
SVQ > SAT -0.038 0.320 0.375 
aMOT >PLGN -0.712 3.846 0.000 
aMOT > SAT -0.142 1.140 0.128 
cMOT>PLGN 0.487 4.501 0.000 
cMOT > SAT -0.134 1.898 0.030 


Key:CQ: content quality; SQ: system quality, SVQ:service quality, IF: instructor factor. SF: student factor, 
aMOT:autonomous motivation, cMOT: controlled motivation, SII:student-instructor interaction, SSI:student-student interaction; 
SCI: student-content interaction. 

Table 1 shows the summary of the moderating effects (path coefficient differences) of e- 
Learning experience among the relationships within the model. There were significant moderating 
effects of e-Learning experience in 13 of the 21 linkages with six skewed towards the 
more experienced students thus, supporting the hypotheses in the following paths/linkages: 
(CQ>PLGN:f-diff=0.697,p<.005);(IF > SAT: B-diff=0.932,p<0.00);(SCI> PLGN:f-diff=0.399, 
p < 0.010); (SF>PLGN:-dif= 0.0.735; p<0.00); (SQDSAT: B-diff=0.932, p< 0.00) and 
(cMOT~> PLGN: B-diff = 0.487; p < 0.00). On the other hand, the moderating effects of e-Learning 
experience was stronger among less experienced students in the following paths (CQ>SAT: B- 
diff= -0.286, p= 0.004); (SFSSAT: B-diff=-0.334,p=0.004); (SSIDSAT: B-diff=-0.268,p=0. 13); 
(SH SAT: B-diff=-0.286,p=0.004); (SQ>PLGN:f-diff=-0.563,p=0.00); (aMOTSPLGN: B- 
diff=-0.712, p=0.00); (CMOT>SAT: B-diff=-0.134,p=0.34). 


These results suggest that in terms of motivation more experienced students appreciate the 
role of controlled motivation more importantly. Also, that low experienced students placed more 
importance on the contents than the technology. This may be ascribed to the maturity of more 
experienced students and longevity of system usage. It may be deduced that the more experienced 
students appreciate more the roles of the instructor, the efficiency of the technology, the quality of 
the materials and interaction with course mates as important factors in e-Learning environments. 
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Additionally, results align with the assertion by Stoel and Hye Lee (2003) regarding the 
significance of longevity of usage to increased appreciation of ease of use and positive attitude to 
technology. Therefore, more attention should be given to the new users of the system to help them 
in actualizing their expectations of the e-Learning environment. More studies are required to better 
understand the role of e-Learning experience in students’ satisfaction. 


Table 2: Results for the significant moderating effects of employment status 


Beta value diff ne p-value 
Relationships Employed — 
Employed vs Not Employed Eibloved y9-Not 
Not Employed Employed 

CQ >PLGN 0.320 2.217 0.014 
CQ > SAT 0.139 0.735 0.232 
IF > PLGN 0.259 1.196 0.117 
IF >SAT -0.145 0.432 0.333 
SAT > PLGN -0.978 5.115 0.000 
SCI > PLGN -0.025 0.158 0.437 
SCI >SAT -0.453 2.089 0.020 
SF >PLGN -0.243 1.456 0.074 
SF >SAT 1.120 4.554 0.000 
SII > PLGN -0.750 2.838 0.003 
SII > SAT 1.127 2.569 0.006 
SQ >PLGN -0.102 0.572 0.284 
SQ > SAT -0.532 1.968 0.026 
SSI > PLGN 0.279 1.603 0.056 
SSI > SAT -0.177 0.705 0.241 
SVQ > PLGN 0.052 0.303 0.381 
SVQ > SAT -0.124 0.442 0.330 
aMOT >PLGN 0.481 2.538 0.006 
aMOT > SAT -0.116 0.575 0.283 
cMOT>PLGN 0.260 1.870 0.032 
cMOT > SAT -0.189 0.970 0.167 


Key: CQ: content quality; SQ: system quality, SVQ: service quality, IF: instructor factor. SF: student factor, aMOT: 
autonomous motivation, cCMOT: controlled motivation, SII: student-instructor interaction, SSI: student-student interaction; SCI: 
student-content interaction. 


From Table 2: It can be seen that there were significant moderating effects of employment 
status with the employed students recording higher path coefficients in the paths: (CQ>PLGN: B- 
diff= 0.32, p= 0.014); (SFOSAT: B-diff= 1.120, p< 0.000); (SU SAT: B-diff= 1.127, p= 0.006); 
aMOT>PLGN: B-diff= 0.481, p= 0.006); and cCMOTPLGN: f-diff= 0.260, p= 0.032).There 
were also significantly negative moderating effects of employment status in the paths: 
(SAT>PLGN: B-diff=-0.978, p< 0.000); (SCIDSAT: B-diff=-0.453, p=0.020); (SHIPLGN: B- 
diff=-0.750, p= 0.003); (SQDSAT: B-diff=-0.532, p=0.026). These results suggest that the “Not 
employed students” had a more significant perception of the system within those relationships. 
And employment students failed to moderate students’ perceived quality towards e-learning 
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satisfaction. The findings paralleled the study by Yuselturk (2009) but conflicted that by Lu and 
Chiou (2010) who reported job status as a moderating factor in interface friendliness and e-learning 
satisfaction. Based on the findings, it may be concluded that the universality of the e-Learning 
environment was emphasised in the study as a system that is open to diverse form of learners. 
Whether experienced, in-experienced, employed, or un-employed. 
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Introduction 


Computational thinking (CT) skills have been specified in many state standards of computer 
science (CS) education (Promote computer science, 2019). They are a set of skills utilizing 
abstraction, algorithmic thinking, decomposition, pattern generalization, and evaluation to solve 
problems (Selby & Woollard, 2013). Educators and researchers have proposed that CT is not 
only used in CS-related fields but also frequently applied in other areas even in our daily lives 
(Barr et al., 2011; Wing, 2006, 2011). 


As a result, investigations have been conducted in recent years regarding CT skill development 
without computers. Researchers deployed non-programming activities in K-12 classrooms 
(Ballard & Haroldson, 2021; Lambert & Guiffre, 2009; Thomas et al., 2019). Those activities 
include games, simulations, picture book reading, or story creation. Results of those studies 
showed a significant improvement of students’ CS/CT skills, especially their basic coding skills, 
such as the definition of variables and use of simple loops and conditionals (Ballard & 
Haroldson, 2021; Grover et al., 2019; Lambert & Guiffre, 2009). 


These findings not only supported the efficacy of non-programming activities in improving 
young students’ CT skills but also encouraged us to further explore young students’ development 
of CT skills prior to coding or robotics programs. 


This study was a part of the learner analysis in the design of a robotics program. In this study, we 
examined the CT skills of middle school students from an underrepresented minority group. Our 
research questions were: 1) did the students’ CT skills in solving gaming problems predict their 
performance of block programming? 2) What other factors, such as gender and prior coding or 
robotics learning experiences, contributed to students’ performance of block programming? 


Methods 
Participants 


Forty-eight students from three classes at a local middle school participated in this study. A 
convenient sampling method was used as the participating students were from the classes where 
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the instructors collaborated with the authors in the robotics program. 42% were girls, and 56% 
were boys. 81% self-identified themselves as African Americans. 4% were Latinos and 
Caucasian, respectively. 67% were in Grade 8, and the rest were from Grade 7. 


Instrument 


We collected and analyzed data from a survey and a quiz, which were delivered at one time in a 
class session. The survey included questions asking students demographic information. The quiz 
included 16 questions from the CTt instrument validated by Roman-Gonzalez et al. (Roman- 
Gonzalez et al., 2016; Roman-Gonzalez et al., 2018). Among them, eight measured students’ 
algorithmic thinking and pattern recognition skills in gaming problems, and 8 measured the same 
skills in block programming problems. Students had 40 minutes to complete the survey and quiz. 


Analysis & Results 


We conducted a multiple regression to identify the predictability of students’ CT skills in solving 
non-programming problems, their prior coding and robotics learning experiences, and their 
gender on their CT skills in solving block programming problems. 


The dependent variable was students’ CT skills in solving block programming problems. It was 
the average of 8 quiz questions measuring CT skills in solving block programming programs. 
Students’ CT skills in solving non-programming problems, their prior coding and robotics 
learning experiences, and their gender were the predictors. Among them, students’ CT skills in 
solving non-programming problems were the average of 8 quiz questions measuring students’ 
CT skills in solving non-programming problems. Students’ prior coding and robotics experiences 
were dummy coded, with 1 meaning Yes and 0 meaning No. Students’ gender was coded as 1 
being boys and 2 being girls. Results of the descriptive statistics of these variables are shown in 
Table 1. 


Table 1. 
Descriptive Statistics 


Mean_| SD N 
CT in Block Programming Problems id 1.14 | 48 


CT in Non-programming Problems pay 1.14 | 48 
Prior Programming Experience mi 38 48 
Prior Robotics Experience AZ 38 48 
Gender .65 48 48 


Results of the multiple regression showed that students’ CT skills in solving non-programming 
problems and their prior robotics learning experience significantly predicted and explained 44% 
variation of their CT skills in solving block programming problems (F(4,43)=8.52, p<.01). 


CT block-programming=.45 *CTnon-programming+.91 *Experiencerobotics 


We did not find a significant impact of students’ prior programming experience and gender on 
their CT skills in solving block programming problems. 


266 


Discussions 


Results of our study showed a significant relationship between students’ CT skills in solving 
gaming and block programming problems. From the instructional design perspective, it supports 
the use of non-programming methods to improve students’ CT skills. At the same time, it 
suggests that students may have developed some CT skills prior to any coding or robotics 
programs. Those skills may be developed from their interactions with real-world problems or in 
prior learning experiences. Therefore, it will benefit students’ understanding of coding if we 
activate their prior knowledge and skills and use the real work examples or their prior knowledge 
to explain the coding algorithms. 


We also found that students’ prior robotics learning experience had a significant impact on their 
CT skills in solving block-programming experiences, but their prior programming learning 
experience did not. We think it may be because robotics provides tangible and visual aids for 
students at young ages to develop an abstract understanding of CT skills. However, studies with 
a large sample size are needed to verify this finding. 


African American students have been reported as underrepresented in secondary CS education 
(Code.org, 2020). In addition, the gender disparity within the underrepresented minority group 
was found to aggravate this inequity (Current perspectives and continuing challenges in cs 
education for u.S. K-12 schools 2020 report, 2020). Results of our post-hoc analysis did not 
show a gender difference in students’ CT skills in solving gaming problems when their prior 
robotics learning experience was taken into account. It could be because only boys were found to 
have prior robotics experiences. This finding endorses the results in the national report. In 
addition, it underscores the necessity to broaden participation for underrepresented minority 
students, especially the female students from that group. We suggest future studies consider the 
impact of students’ prior experiences when examining the gender difference. 
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Abstract: With the development of education information, a new round of scientific and 
technological changes and challenges have set new goals for talent cultivation and education 
innovation and put forward new requirements for the analysis of learners' characteristics. The use 
of user portrait technology in education can help teachers to keep pace with the development of 
learners and provides a way to promote the future development of students' learning. And data 
collected by electronic schoolbags in real time make data-based analysis feasible. This paper 
mainly uses data-based learning analysis method to analyze and mine the data of a fourth-grade 
elementary school. Firstly, the pre-processed data is descriptively tested to see if it conforms to 
the normal distribution. After that, the student's grades are standardized and clustered. Secondly, 
the processed data is used to map the student's comprehensive situation radar map. Based on the 
results of radar map and cluster analysis, targeted suggestions are put forward for learning and 
teaching. This study provides a reference model and a new thinking dimension for data-based 
learner feature analysis and the research results have theoretical value and practical significance. 
Keywords: Analysis of learner characteristics; User portrait; Electronic schoolbag 


1 Introduction 


Since the 21st century, education information reform has been unanimously recognized in 
the education field. Subsequently, the "Bring Your Own Device (BYOD)" campaign, which allows 
every student to obtain and use a mobile device, began to be promoted (Liu Bin et al., 2016). 
Through these portable mobile devices, students can read various electronic textbooks and online 
courses and other resources needed for learning, which is conducive to the development of one- 
to-one digital learning. The electronic schoolbag is a new technological product born with the 
development of science and technology since the 21st century. E-schoolbags have been 
experimenting in the field of foreign education a long time ago. However, for various reasons, the 
development of e-schoolbags in China has not kept pace with foreign countries and started late 
(Tong Hui et al., 2016). It was not until 2013 that e-schoolbags finally started to be tested in China 
(Wang Yujie, 2018). At present, due to the characteristics of the e-schoolbag itself, its scope of 
application is not wide. E-bookbag trials have been conducted almost only in schools and areas 
with better educational resources and higher quality of teaching. The electronic schoolbag uses an 
electronic terminal to integrate the students' books. It can change the traditional classroom format, 
not only the carrier of teaching, but also the teaching model to a certain extent (Fan Minsheng et 
al., 2017). Electronic schoolbags are an important means of communicating traditional school 
education and education informatization. It creates conditions for the collection of learner data, 
which in turn provides strong support for the development of data-based learning analysis (Zheng 
Liging, 2014). 

This research is mainly devoted to the analysis of learner characteristics based on data, and 
at the same time, it provides a reference model and some new thinking dimensions for the analysis 
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of learner characteristics based on data in the background of information technology. Therefore, 
this research uses a series of data from e-schoolbags to analyze the characteristics of learners and 
make suggestions. 


2 Literature Review 


2.1 Learning Analysis 

As an emerging field of learning analysis, its development is closely related to information 
technology and educational intelligence. Learning analysis uses data and models to predict the 
future state of learners and discover potential problems through analysis (Liu Qingtang et al., 
2017). Learning analysis reports can help to better understand learners’ learning styles, learning 
habits, and cognitive structure, and other information, so it helps teachers to prescribe the right 
medicine and develop personalized learning plans for students (He Kekang, 2016). In addition, 
through learning analysis, a comprehensive observation and recording of the entire learning 
process of learners can be used to grasp the learning situation of students in real time, which can 
help teachers make timely adjustments to their own teaching plans and teaching strategies. 
Nowadays, to measure, collect and analyze learners and learning environment, and get 
corresponding reports, and then use these reports to optimize the learning process of learners and 
the learning environment in which they are located. It is a technology for learning analysis. 
Accepted definition (Wu Yonghe et al., 2013; Siemens, 2011). 

Learning analysis has attracted the attention of domestic scholars since the Horizon Report 
released in 2011. Gu Xiaoging and others define learning analysis as: using different analysis 
methods and data models to explain the data related to learner information, and then explore the 
learning process and laws of learners; or interpret learners’ learning performance based on data 
and provide them with corresponding feedback Thereby promoting more effective learning (Gu 
Xiaoqing et al., 2012). Learning analysis is based on the massive omni-directional data generated 
by students in the learning process to carry out data analysis and mining, using visual means to 
present the results of data analysis and mining, and then teachers and teaching administrators can 
carry out data analysis and mining based on the obtained predictive model The prediction of 
students' various learning behaviors. In this way, it is helpful to find risk learners and intervene in 
time. In addition, it can also push personalized learning resources and design personalized learning 
paths for students. Therefore, learning analysis can be regarded as achieving teaching students in 
accordance with their aptitude. Technical support for the purpose. 

2.2 Learner Characteristics Analysis 

The new round of technological changes and challenges has put forward new goals for 
talent training and educational innovation and has also put forward new requirements for the 
analysis of learner characteristics. "Taking the learner as the center" is a major purpose of the new 
situation classroom under the guidance of current constructivism (Yu Jiajun, 2015). Compared 
with the traditional classroom, the role of teacher and student are changed, so that the learner can 
change from a passive position to an active position and become a meaning builder. At the same 
time, teachers have also changed their previous roles, becoming instructors and facilitators of 
learning. Therefore, students are extremely important for the ability to clearly understand their 
own original cognitive structure and to actively construct the meaning of relevant knowledge based 
on their own experience. Learner analysis has become an important part of it. Based on this 
background, many scholars and researchers have conducted in-depth explorations on learning 
analysis in the new educational information environment (Xu Qi et al., 2021; Chen Changsheng et 
al., 2020; Sabine Seufert et al., 2019). With the development of the education field, the content 
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and dimensions of learning analysis are also constantly changing. On online learning platforms 
(such as the MOOC platform), thanks to the comprehensiveness and convenience of platform data 
acquisition, many meaningful research results have emerged, which has promoted the development 
of these platforms. However, due to the inconvenience of the acquisition and collection of offline 
course data, there are still problems in applying similar learning analysis techniques to traditional 
classrooms (Huang Qin, 2017). 

Data mining on the acquired education data through learning analysis technology can 
provide strong support for the development of education and teaching. Learning analysis can cover 
the entire process from diagnostic evaluation to formative evaluation to summative evaluation, 
providing powerful support for schools, teachers, students, and parents to learn about teaching, 
learning, and results, thereby providing guidance for promoting learning and improving 
performance. Therefore, how to use the existing experience in traditional school education to carry 
out the analysis of learning that meets the new requirements of today, to promote the development 
of classroom education, is a problem worthy of consideration and exploration. 

2.3 Learner Portrait 

As far as the current situation is concerned, at the practical level of the education field, the 
big data user profile technology has not been tested in the true sense (Xu Yan, 2017). Learner 
portrait is the multi-dimensional quantification of learner's internal or external characteristics. It is 
an abstract student model obtained based on data analysis and mining. Learner portraits outline 
the image of learners through the modeling of related data, restore students’ various characteristics, 
motivation levels and other potential attributes, and then understand learners' behavioral 
tendencies and needs, and derive learning laws hidden in many learners. Carrying out the analysis 
of learner characteristics in different dimensions for different learning scenarios can improve the 
design of learning platform, improve the learning recommendation system and teaching strategies 
(Xiao Jun et at., 2019). Most of the foreign researchers carry out research by dividing the roles of 
learners by certain characteristics, which include learners' learning motivation, cognitive level, and 
learning attitude. Scholars such as Wataru Takahashi provide corresponding services to different 
learners based on the motivation composition, self-efficacy cycle, and service incentive effects of 
different roles. Takahashi et al., 2014). Foreign scholars have also conducted in-depth research on 
how to build learner portraits. The goal of establishing a learner's profile is to focus on the 
implementation of learning needs, motivations, preferences, etc., and present the main 
characteristics of learners. Scholars such as Debbie Holley divide students into different risk levels, 
predict the learners' use of risk models, and then implement corresponding interventions. 
Therefore, accurately describing learning ability, providing guidance for teachers to carry out 
teaching, and promoting the improvement of professional skills are the important values of 
establishing learner profile. 

With the continuous in-depth development of the concept of learner portraits, domestic 
scholars have gradually begun corresponding research. Through research, some scholars believe 
that the establishment of learner profile in a ubiquitous learning environment is an important part 
of the design of MOOC resources, and the evaluation of learning effects can also be achieved 
through learner profile (Wang Xiaofang et al., 2019). The significance of learner profile 
technology in personalized teaching has also been confirmed by many empirical studies conducted 
by domestic scholars based on experiments (Chen Haijian et al., 2017; Tang Yewei et al., 2019). 

In summary, as the application of user portrait technology in education, learner portrait 
technology plays an important role. The learner profile technology can not only make precise 
positioning according to the different characteristics of learners, but also can be further used in the 
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optimization link of teaching design to provide targeted support for learning, to better serve 
personalized learning. However, at present, the research on learner profile technology still focuses 
on how to build models and how to apply them, and there has not been an overall description of 
the process of learner profile building. 


3 Method 


3.1 Research Path 

In this research, EXCEL, RStudio, and SPSS are used to model with learner characteristics 
analysis method based on data. First, the obtained student performance data is standardized, and 
then various indicators are used for cluster analysis. Then, the standardized student performance 
is non-linearly transformed, and a radar chart is drawn based on the standardized student 
performance, and the radar chart is used to perform a single Analysis of student performance to 
promote the overall development of students. 


3.2 Data Sources 

A certain elementary school aims to promote the application of information technology in 
education and has carried out a long period of in-depth exploration in the use of electronic 
schoolbags to carry out teaching. The "electronic schoolbag" used by the fourth-grade students in 
this elementary school has many special functions, which can enhance classroom interaction and 
produce better teaching results. The emergence of electronic schoolbags has created conditions 
for the development of data-based learning analysis in school education, and the use of electronic 
schoolbags in the fourth grade of a primary school is providing a good case study for data-based 
learning analysis in school education. Thus, this research started. The daily teaching activities in 
the fourth grade of a primary school are basically carried out using electronic schoolbags. 
Through the electronic schoolbag, teachers use the online course platform to present teaching 
content, collect teaching materials, interact with students, publish assignments, and test 
information, and carry out evaluation activities; students use the online course platform to 
conduct group interactive discussions, communicate with teachers, and obtain teaching Activities 
such as content, completion of homework and quizzes. In this process, a large amount of student 
data was generated. These data include student PAD usage, classroom interaction, evaluation 
activities, evaluation results statistics, knowledge mastery, statistics on the number of teacher 
lectures, etc. Among them, some data about the evaluation activities are incompletely recorded, 
students' knowledge mastery in some time periods, and individual student data are missing. 

The data selected in this study comes from the data in an electronic schoolbag used by 29 
fourth-grade students in a primary school. The daily teaching tasks of this elementary school are 
basically carried out through electronic schoolbags. It can be considered that the acquired 
electronic schoolbag data is a good and comprehensive record of the situation of the fourth-grade 
students in a certain elementary school. 


3.3 Data Processing 
3.3.1 Standard Score 

The standard score is a relative status measure, which is derived based on the original score. 
The standard score can represent the relative position of the original score in all the scores in which 
it is located. The standard score is based on the standard deviation of a group of scores, and the 
average of the number of components is used as a reference. It can show that the original score is 
several standard deviations away from the overall average in the whole, and then it reflects the 
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relative position of this original score in the overall data. Compared with the original score, the 
standard score can more intuitively reflect the meaning of the obtained data. The standard score 
can reflect the position of the student in the overall. As a result, the same standard score can still 
be regarded as equivalent even if the test is different. This also provides a method for visually 
comparing scores in different test contexts. From this, after converting the learner's raw scores into 
standard scores, it will be more comparable, especially in different times, different subjects, and 
different types of tests. It will be more reasonable and fairer to use standardized student scores in 
learner analysis. 
3.3.2 Cluster Analysis 

Cluster analysis is a process of classifying and combining individual abstract individuals 
in a whole and combining individuals with similar characteristics to form classes. Cluster analysis 
is an important analysis method. The clustering effect depends on two factors: the method of 
measuring the distance and the clustering algorithm. K-means clustering algorithm is a commonly 
used method of clustering analysis. Its approach can be expressed as selecting K initial centroids 
at the beginning. The so-called centroid is the average vector of all observations in a class. The 
initial centroid can be selected randomly, each centroid is a class, then each observation is assigned 
to the centroid closest to it, and a new class is formed with the centroid, and finally the centroid of 
each class is recalculated and repeated until the centroid does not occur. When changing or 
reaching the maximum number of iterations. This research mainly uses K-means clustering 
algorithm to analyze student data. 
3.3.3 Non-linear transformation 

The student's standard score z obtained after conversion is a standard normal distribution 
N(, 0). In order to facilitate the use of the obtained standard scores in subsequent analysis, non- 
linear conversion processing is performed on them. 


: : : 1 = 
The nonlinear conversion formula used is z’ = —tan7~! z+ 0.5. 
TT 


The transformation has the following characteristics: 

(1) Convert the infinite interval from negative infinity to positive infinity into a finite 
interval from 0 to 1; 

(2) Convert the original average value of the standard score (z) from 0 to an average value 
of 0.5; 

(3) The function obtained after transformation has good linearity near the average value, 
and the more it deviates from the average value, the stronger its compressibility. 
3.3.4 Radar chart 

The radar chart is a graphical method of displaying multivariate data in the form of a two- 
dimensional chart of three or more variables on an axis starting from the same point. The radar 
chart is composed of a series of equidistant concentric circles. Each of the concentric circles 
corresponds to a different value, the closer to the center of the circle, the smaller the value, and the 
greater the deviation from the center of the circle, the larger the value. The evaluation index of the 
radar chart is represented by multiple axes drawn from the center of the circle. The value 
corresponding to each index obtained is processed and marked on the index axis according to a 
certain ratio, and then the marked points obtained by these marks are aligned. If you are connected 
to each other, you can get a radar chart that reflects each evaluation index of the sampled 
book.Radar chart plays a very good role in both qualitative analysis and quantitative analysis. 
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4 Results 


4.1 Overall Characteristic of Learners 

Use SPSS to explore the PAD use time and reading time in the obtained student data, and 
judge whether it obeys the normal distribution. The results are shown in Table 1. The absolute 
values of the skewness coefficient and kurtosis coefficient of the two indicators are both less than 


1, which can be regarded as approximately obeying a normal distribution. 
Table 1. Statistical results of PAD use time and Reading time 


Statistics Standard Error 
PAD use timeReading time _PADusetime Reading time 
Mean 1157.497 39.735 131.4139 6.2917 
95% confidence interval Lower limit 889.114 26.886 
of the mean Upper limit 1425.880 52.585 
5% trimmed mean 1109.367 37.654 
Median 827.000 28.000 
Variance 535358.163 1227.140 
Standard deviation 731.6817 35.0306 
Minimum 340.0 2.4 
Maximum 2889.7 115.6 
Scope 2549.7 113.2 
Interquartile range 1061.0 59.5 
Skewness 894 .904 421 421 
Kurtosis -.156 -.478 821 821 


After sorting out the obtained student's original scores, select the entire class of students in 
a certain test, and standardize the student scores in the following way. 
(1) Obtain the scores of n students in the test in order of x1, X2...Xn, calculate the average 
Oe ges 
score of the whole class of students who participated in the test pu: p = ae 


(2) Calculate the standard deviation o of the obtained scores of n students in the test: o = 


UTi-B)?, 
n 9 


(3) According to the standard score conversion formula, the original scores obtained by 
each student in the test are converted into standard scores for easy comparison by calculation. The 
formula is Zz = me 

Some of the final processing results are shown in Table 2. 

Table 2. Standardization 


Student Chinese Math English 
Chen 0.104 0.482 -0.009 
Dai 0.297 -0.184 -0.460 
Du 0.406 0.251 0.298 
Fengl -0.445 0.372 0.138 
Zhang3 0.288 0.149 0.298 


Standardize students’ classroom performance, teacher evaluation, pad use time, and 
reading time data. Select five indicators: performance indicators, evaluation indicators, interaction 
indicators, PAD use time, and reading time, and use SPSS to perform cluster analysis. It is divided 


into 5 categories, Table 3 shows the cluster analysis results. 
Table 3. Cluster center 
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Cluster 


1 2 3 4 5 
performance 1212 -.1421 -.0464 1117 -.2021 
interaction -.0923 .0337 -.1015 .1630 -.4146 
evaluation -.1005 1645 -.1369 1143 -.4125 
PAD use time = -.2390 -.2416 .1967 .2334 2457 
reading time -.1174 -.1465 .2163 -.0561 -.1798 


RadViz radar chart can complete multi-dimensional data visualization tasks and is suitable 
for visualization of cluster analysis results. It is based on the basic spring pressure minimization 
algorithm often used in complex network analysis. The spring algorithm puts a series of nodes in 
the same plane, and then assumes that each data set uses a spring to connect to each node. 
Therefore, these nodes will move due to the existence of springs and stop when all the elastic 
potential energy of the entire system reaches a minimum. In the RadViz radar chart, different 


1.00 — 2 
e 


eeee 


colors are used to identify different classes. Figure 1 shows the drawing result. 
Figurel. RadViz radar chart 

The first category of cluster analysis results contains 11 students. These 11 students have 
good performance in performance indicators, but most of them do not perform well in classroom 
interaction, teacher evaluation, PAD use time and reading time indicators. Through analysis, we 
can clearly know that they are all able to achieve good results, but they do not tend to use PAD as 
a learning tool and prefer traditional book learning, nor do they like to interact with teachers in the 
classroom. From a long-term perspective, although such students can better complete their 
academic tasks through independent learning, they should still interact and communicate more 
with teachers, to help them discover new problems in their communication with teachers. At the 
same time, even if such students use less e-bookbags, they can still achieve better results. Teachers 
can still guide these students to use modern tools to help them learn, so that students can realize 
the power of the new learning tool of e-bookbags. The convenience provided for their study. 

The second category of cluster analysis results includes 6 students. These 6 students have 
good performance in classroom interaction and teacher evaluation indicators, but most of them do 
not perform well in performance indicators, PAD use time and reading time indicators. This shows 
that although they actively participate in the interaction in the classroom, they can also get good 
evaluations from the teachers in the classroom, but they still cannot achieve satisfactory results. 
This kind of students are also more common in the classroom. Although they are active in the 
classroom, they may not get a good grade due to inappropriate learning methods and methods. 
Therefore, helping such students find a learning method that suits them is one of the key points to 
improve their academic performance. In addition, teachers can also consider starting from the use 
of PAD and consider whether these students are unable to achieve the expected results compared 
with other students, whether it is because they did not maximize the help effect of e-schoolbags 
on learning. Correct guidance to improve student performance. 
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The third category of cluster analysis results includes 5 students. These 5 students have 
outstanding performance in PAD use time and reading time indicators, but most of their 
performance indicators, classroom interaction indicators and teacher evaluation indicators are not 
optimistic. For such students, it should be considered whether they have not properly played the 
role of PAD, an e-schoolbag. When these students use e-schoolbags, they are likely to use it as an 
entertainment tool and fail to achieve the purpose of e-schoolbags to help and promote learning. 
These students should be guided in a timely manner, and they should be guided to use e-schoolbags 
to help their learning, instead of using them as a novel pastime and play tool. The real significance 
of introducing e-schoolbags is to use the value of e-schoolbags. 

The fourth category of cluster analysis results contains 8 students, these 8 students have a 
good performance in other aspects except for the lower PAD reading time indicator. This shows 
that these 8 students can better promote their own learning through the interaction with the teacher 
in the classroom, can also get a better evaluation from the teacher, and can also use the electronic 
schoolbag to complete the learning task better. In addition, these 8 students can consider using e- 
bookbags to read more, study textbooks, content outside the classroom, enrich their knowledge 
content, and achieve better performance. 

The fifth category of cluster analysis results includes 1 student. This student has 
outstanding performance on the PAD usage time indicator, but the performance on other indicators 
is not optimistic. This student is like the 5 students in the third category, and it should be 
determined in a timely manner whether he has correctly played the role of the e-schoolbag and 
actively guided. 

From the above analysis, different students have different attitudes and situations towards 
learning with electronic schoolbags. However, it is not difficult to find from the results that the 
current students' acceptance and use of e-schoolbags need to be improved, and e-schoolbags still 
do not fully exert their value in helping to learn to a certain extent. 

Through cluster analysis, 31 students are divided into 5 categories according to the five 
dimensions of performance indicators, evaluation indicators, interaction indicators, PAD usage 
indicators, and PAD reading time indicators, which helps to intuitively and clearly grasp the 
various Dimensional performance, and promptly discover possible problems in the learning 
process of students, so as to help teachers to carry out personalized teaching and prescribe the right 
medicine, so as to guide students to better aspects and promote the overall development of students. 
In addition, clustering analysis based on data can not only improve the efficiency of teachers in 
understanding the situation of students, but also more scientifically obtain the situation of students 
from the observation and feeling of teachers in traditional classroom teaching and has a good 
guidance for the development of teaching. significance. 


4.2 Individual Characteristic of Learners 


Table 4 shows some of the processing result after non-linear transformation. 
Table 4. Processing result after non-linear transformation 


Student Chinese Math English 
Chen 0.533 0.643 0.497 
Dai 0.592 0.442 0.363 
Du 0.623 0.578 0.592 
Feng 0.367 0.613 0.544 
Dai 0.589 0.547 0.592 
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Do the same processing with the obtained data of student classroom performance, teacher 
evaluation, pad use time, and reading time, and select indicators Chinese score, Chinese 
evaluation, Chinese interaction, math score, math evaluation, math interaction, English score, 
English evaluation, English interaction, PAD use time, reading time 11 indicators, use RStudio to 
draw a radar chart. 

When drawing, convert the 11 selected indicators into 11 corresponding dimensions in the 
radar chart. According to the results of data processing, the more the indicator point deviates from 
the central origin, the better the student’s performance on the indicator. From the perspective of 
the area of the closed figure enclosed by each indicator point of the radar chart, if the area of the 
closed figure is larger, the overall situation of the students is better. 


4.2.1 Case Study 1 

Figure 2 is a radar chart of Zhang's comprehensive situation. From the figure, we can see 
that Zhang’s mathematics performance indicators, mathematics evaluation indicators, English 
performance indicators, English interaction indicators, and PAD use time indicators are all close 
to the maximum value of the class. Reading time indicators, language interaction indicators, and 
English evaluation indicators It is higher than the average of the whole class, the Chinese 
performance index and the Chinese evaluation index are close to the average of the whole class, 
and the mathematics interaction index is lower than the average of the whole class. 


—* Zhang 
—* Minimum 


—* Maximum 
Evaluation 


Figure2. RadViz radar chart of Zhang's comprehensive situation 

(1) Teaching interaction. By observing the three interactive indicators of Zhang’s Chinese, 
Mathematics, and English, we can find that Zhang can interact well with teachers in Chinese and 
English classes, but he did not interact with teachers to a similar degree in math class. This shows 
that in terms of teaching interaction, Zhang's situation is better, but he still needs to pay attention 
to his interaction in the mathematics class and find the reasons for his low mathematics interaction 
indicators, such as the tendency to teaching methods. 

(2) Technical application. From the two technical application indicators of Zhang (PAD 
use time and reading situation), we can find that Zhang’s PAD use time indicator is close to the 
highest value of the class, and his reading time indicator is also higher than the average of the 
class. This shows that Student Zhang can use the electronic schoolbag to study well and complete 
the tasks assigned by the teacher. 

(3) Subject analysis. From the radar chart of Zhang’s comprehensive situation, we can find 
that Zhang’s English scores and math scores are close to the highest value in the class, indicating 
that Zhang has a good performance in these two subjects. In contrast, Zhang’s Chinese 
performance is a bit weak, and his Chinese performance indicators are only close to the average 
Chinese performance of the whole class. As a Chinese teacher, he should pay more attention to its 
internal reasons, such as whether the learning method is inappropriate, the learner Zhang himself 
can also improve his language performance through self-reflection. 
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(4) Teacher evaluation. Among the three teacher evaluation indicators of Zhang, the 
mathematics evaluation index tends to be the highest in the class, and the English evaluation index 
is also higher than the average of the class, indicating that Zhang can be highly recognized by the 
teachers in these two subjects. However, Zhang's language evaluation index is only near the 
average of the class. Based on the analysis of Zhang's language performance indicators and 
language interaction indicators above, there is still a lot of room for improvement in Zhang's 
performance in language subjects. 


4.2.2 Case Study 2 

Figure 3 is a radar chart of Teng's comprehensive situation. From the radar chart, we can 
understand that Teng’s PAD usage time and English performance indicators tend to be the highest 
in the class. In addition, Teng’s Chinese performance indicators, language interaction indicators, 
language evaluation indicators, and mathematics scores the indicators, mathematical interaction 
indicators, mathematics evaluation indicators, English interaction indicators, English evaluation 
indicators and PAD reading time indicators all reach the average of the class or even tend to the 
lowest value of the class. 


Figure3. RadViz radar chart of Teng's comprehensive situation 

(1) Teaching interaction. Observing and analyzing the interactive indicators of Teng's three 
subjects, it can be found that these three interactive indicators of Teng are close to the lowest value 
of the whole class. From this we can infer that student Teng does not like to participate in learning 
by interacting with teachers in class. In this regard, teachers may consider guiding Teng. When 
considering Teng, it is for reasons of character to guide him to participate more actively in the 
classroom. 

(2) Technical application. From the use of Teng’s PAD and reading time indicators, we 
can find that Teng’s PAD is used more frequently, and it is understood that most of Teng’s other 
indicators are basically the lowest in the class, indicating that Teng Although students often use 
the electronic schoolbag as a tool, they may not give full play to the value and function of the 
electronic schoolbag. Based on this problem, the teacher should provide timely guidance to achieve 
the purpose of the electronic schoolbag to promote learning. 

(3) Subject analysis. Observing Teng's Chinese, math, and English performance indicators, 
we can find that, except for the English performance indicators tending to the highest value of the 
class, the other two indicators are close to the lowest value of the class. This shows that Teng can 
achieve good results in English subjects, but his performance in Chinese and mathematics subjects 
is not good. This may be due to Teng's preference for English subjects. In this regard, teachers 
should start from the perspective of the content of each subject to help Teng understand the 
meaning of each subject to himself, to stimulate Teng's inner motivation for learning other subjects 
and promote the overall development of his various subjects. 
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(4) Teacher evaluation. Analyzing the teacher evaluation indicators of Teng in the three 
subjects of Chinese, mathematics, and English, we can find that Teng has not been able to get a 
good evaluation of the teachers in these subjects. Comparing and analyzing Teng's teaching 
interaction indicators and subject performance indicators, we found that they are almost 
corresponding. Therefore, teachers need to consider whether this is caused by Teng's learning 
interest or lack of learning motivation. After investigating and clarifying the reasons, they will 
prescribe the right medicine to help Teng become a better learner. 

4.2.3 Comparative Analysis of Two Cases 

By observing the radar chart of the two students, we can also observe the overall situation 
of the students through the area of the closed figure, that is, the larger the area of the closed figure, 
the better the overall situation of the students. Comparing the radar charts of the two students, we 
can easily find that the two students performed similarly on the English performance indicators 
and the PAD usage time indicator. 

Judging from the three performance indicators alone, Zhang’s performance in each subject 
has achieved a more balanced development, while Teng’s performance in Chinese and 
mathematics is still very lacking in addition to English. This is easily reminiscent of the law of 
short boards. The amount of water that a wooden barrel can hold depends on the shortest wooden 
board of the wooden barrel, not the longest wooden board of the wooden barrel. Each of us is like 
a wooden barrel, with its own length and shortness. The longer plank can naturally help us to have 
some outstanding performance, but in the end our development will be largely determined by the 
shortest plank, so We should work hard to make up for the shortcomings. As a student, you should 
also achieve a balanced development of all subjects to promote your own outstanding 
development. Through the radar chart, students can clearly and intuitively find their shortcomings 
and try to make up for them. 


5 Discussion 


By drawing and analyzing the radar chart of the overall situation of students, it is helpful 
for teachers to grasp the situation of all students from the overall situation and formulate teaching 
strategies in the general direction. Analyzing the radar chart of a single student will help teachers 
better carry out personalized teaching, understand the strengths and weaknesses of students, and 
prescribe the right medicine to promote the all-round development of students. By observing their 
own data analysis radar chart, students can intuitively understand their own development in various 
aspects, more immediately understand their own strengths and deficiencies, guide students to learn 
to check for missing vacancies, and promote students to form a self-reflection, self-promotion, and 
self-improvement Virtuous circle. 

Using the large amount of student data generated in real time during the use of the 
electronic schoolbag to perform corresponding learning analysis and generate student 
characteristic analysis is of broad significance for the development of further teaching and the 
progress of student learning. From a practical point of view, the analysis of learner characteristics 
based on data has the following significance. First, for students, obtaining learner characteristics 
analysis based on data analysis will help students to understand their real-time strengths and 
weaknesses more intuitively in time, forming a virtuous circle of self-reflection and self- 
promotion. Second, as far as teachers are concerned, obtaining real-time characteristics analysis 
results of students can help teachers better meet the needs of providing personalized teaching, 
grasp the strengths and weaknesses of students, and then set personalized development goals to 
guide students toward complete and comprehensive development. Third, for parents, compared to 
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simply knowing the learning results of students through test scores, feature analysis based on 
student learning data can provide parents with a measure of measurement. From a theoretical point 
of view, this research can provide a reference model for data-based learner characteristics analysis 
under the background of information technology. This research involves the use of a series of data 
including statistics on students’ classroom performance, statistics on the mastery of students’ 
knowledge points, statistics on the use of students’ electronic schoolbags, statistics on student 
performance trends, completion time and completion statistics of student homework and tests, etc., 
to analyze and generate a series of data. Series learner characteristics. This data-based learner 
characteristic analysis mode can provide a reference for the development of learning analysis, and 
it also provides some new thinking dimensions about learner characteristic analysis. 

There are still shortcomings in this research. Due to the large and complex data obtained, 
there is not much data left after actual screening, especially the data of some individual students 
have serious lacks, and some subjects and knowledge points cannot be determined, so students 
cannot be determined. The individual performance in each period is compared longitudinally. In 
addition, the originally obtained student performance data is not only Chinese, mathematics, and 
English, but also student science and other subjects. However, due to the small number of these 
subjects and the large time interval, they are not suitable for analysis. Therefore, this study only 
analyzed the three subjects of Chinese, Mathematics, and English. 
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